CUPTI Sample Code problem

fwd20001101 · June 24, 2025, 8:09am

Im trying to leran cupti these dayse,however when im trying to compile and run CUPTI sample codes,i found that kernel seems not running on my device.
For example when im running sample code :callback_event,i got these output messages:

fangweidong@syh-computed:~/fwd/cupti-tutorial/callback_event$ ./callback_event
Usage: ./callback_event [device_num] [event_name]
CUDA Device Number: 0
CUDA Device Name: NVIDIA RTX A4000
Compute Capability of Device: 8.6
Warning: Sample unsupported on Device with compute capability > 7.2. Waiving test.

My CUDA version is 12.8,so anyone knows what wrong here?Big Thanks!!!

veraj · June 24, 2025, 8:15am

Hi, @fwd20001101

Warning: Sample unsupported on Device with compute capability > 7.2. Waiving test.

---------- This means this sample do not support on your GPU. Please choose other samples to check.

fwd20001101 · June 24, 2025, 8:34am

So fast，thanks for your reply！After browsing the forum to find similar problems，i got these conclusions below，can you help me judge am i right?
Conclusion:
CUPTI APIs like callback API/activity API are designed and used based on some old gpus(which compute capability is below 7.5),so while running these APIs’ sample codes errors may occur.

So which CUPTI APIs may support my device and can achieve the same kernel profiling effect like callback/activity API?

veraj · June 24, 2025, 8:45am

Hi, @fwd20001101

You can refer APIs usage in 5. Modules — Cupti 12.9 documentation

Also we provided several profiling samples like callback_profiling/autorange_profiling etc

mjain · June 24, 2025, 10:07am

CUPTI provides several APIs for tracing and profiling, but not all APIs are supported on all GPU architectures. For example, the CUPTI Event API and CUPTI Metric API are not supported on any GPU architecture after Volta i.e. compute capability 7.5 or higher. Consequently, the sample callback_event which is based on the CUPTI Event API fails to run on Ampere GPUs, such as the NVIDIA RTX A4000 with compute capability 8.6.

And the CUPTI Callback API and the CUPTI Activity API are designed to run on all CUDA supported devices.

For more details, please refer to the following sections in the CUPTI documentation:

fwd20001101 · June 24, 2025, 11:27am

But if callback API and Acitivity API are designed on all CUDA supported devices，when im compiling and running activity_trace in sample code,i got this as output:

Device Name: NVIDIA RTX A4000
DEVICE NVIDIA RTX A4000 (0), capability 8.6, global memory (bandwidth 427 GB/s, size 15992 MB), multiprocessors 48, clock 1560 MHz
OVERHEAD [ 152448106, 152450957 ] THREAD id 1787117568
DRIVER cbid=677 [ 155580247 - 155583215 ] process 2743138, thread 1787117568, correlation 1
DRIVER cbid=677 [ 155583469 - 155583829 ] process 2743138, thread 1787117568, correlation 2
DRIVER cbid=677 [ 155583976 - 155584272 ] process 2743138, thread 1787117568, correlation 3
DRIVER cbid=677 [ 155584408 - 155584524 ] process 2743138, thread 1787117568, correlation 4
DRIVER cbid=677 [ 155584597 - 155584879 ] process 2743138, thread 1787117568, correlation 5
DRIVER cbid=677 [ 155584953 - 155585202 ] process 2743138, thread 1787117568, correlation 6
DRIVER cbid=677 [ 155585346 - 155585528 ] process 2743138, thread 1787117568, correlation 7
DRIVER cbid=677 [ 155585598 - 155585980 ] process 2743138, thread 1787117568, correlation 8
DRIVER cbid=677 [ 155586114 - 155586332 ] process 2743138, thread 1787117568, correlation 9
DRIVER cbid=677 [ 155586406 - 155586760 ] process 2743138, thread 1787117568, correlation 10
DRIVER cbid=677 [ 155586838 - 155587076 ] process 2743138, thread 1787117568, correlation 11
DRIVER cbid=677 [ 155587206 - 155587385 ] process 2743138, thread 1787117568, correlation 12
DRIVER cbid=677 [ 155587452 - 155587618 ] process 2743138, thread 1787117568, correlation 13
DRIVER cbid=677 [ 155587691 - 155588009 ] process 2743138, thread 1787117568, correlation 14
DRIVER cbid=677 [ 155588074 - 155588347 ] process 2743138, thread 1787117568, correlation 15
DRIVER cbid=677 [ 155588420 - 155588587 ] process 2743138, thread 1787117568, correlation 16
DRIVER cbid=677 [ 155588724 - 155588880 ] process 2743138, thread 1787117568, correlation 17
DRIVER cbid=677 [ 155588952 - 155589149 ] process 2743138, thread 1787117568, correlation 18
DRIVER cbid=677 [ 155589222 - 155589319 ] process 2743138, thread 1787117568, correlation 19
DRIVER cbid=677 [ 155589381 - 155589886 ] process 2743138, thread 1787117568, correlation 20
DRIVER cbid=677 [ 155589959 - 155590224 ] process 2743138, thread 1787117568, correlation 21
DRIVER cbid=677 [ 155590292 - 155590439 ] process 2743138, thread 1787117568, correlation 22
DRIVER cbid=677 [ 155590568 - 155590803 ] process 2743138, thread 1787117568, correlation 23
DRIVER cbid=677 [ 155590877 - 155591034 ] process 2743138, thread 1787117568, correlation 24
DRIVER cbid=677 [ 155591166 - 155591335 ] process 2743138, thread 1787117568, correlation 25
DRIVER cbid=677 [ 155591409 - 155591541 ] process 2743138, thread 1787117568, correlation 26
DRIVER cbid=677 [ 155591607 - 155591807 ] process 2743138, thread 1787117568, correlation 27
DRIVER cbid=677 [ 155591938 - 155592090 ] process 2743138, thread 1787117568, correlation 28
DRIVER cbid=677 [ 155592162 - 155592247 ] process 2743138, thread 1787117568, correlation 29
DRIVER cbid=677 [ 155592319 - 155592409 ] process 2743138, thread 1787117568, correlation 30
DRIVER cbid=677 [ 155592481 - 155592605 ] process 2743138, thread 1787117568, correlation 31
DRIVER cbid=677 [ 155592679 - 155592806 ] process 2743138, thread 1787117568, correlation 32
DRIVER cbid=677 [ 155592965 - 155593117 ] process 2743138, thread 1787117568, correlation 33
DRIVER cbid=677 [ 155593190 - 155593331 ] process 2743138, thread 1787117568, correlation 34
DRIVER cbid=677 [ 155593406 - 155593561 ] process 2743138, thread 1787117568, correlation 35
DRIVER cbid=677 [ 155593834 - 155593987 ] process 2743138, thread 1787117568, correlation 36
DRIVER cbid=677 [ 155594060 - 155594218 ] process 2743138, thread 1787117568, correlation 37
DRIVER cbid=677 [ 155594290 - 155594472 ] process 2743138, thread 1787117568, correlation 38
DRIVER cbid=677 [ 155594539 - 155594782 ] process 2743138, thread 1787117568, correlation 39
DRIVER cbid=677 [ 155594856 - 155595058 ] process 2743138, thread 1787117568, correlation 40
DRIVER cbid=677 [ 155595190 - 155595262 ] process 2743138, thread 1787117568, correlation 41
DRIVER cbid=677 [ 155595335 - 155595491 ] process 2743138, thread 1787117568, correlation 42
DRIVER cbid=677 [ 155595557 - 155595689 ] process 2743138, thread 1787117568, correlation 43
DRIVER cbid=677 [ 155595813 - 155595895 ] process 2743138, thread 1787117568, correlation 44
DRIVER cbid=677 [ 155595968 - 155596117 ] process 2743138, thread 1787117568, correlation 45
DRIVER cbid=677 [ 155596191 - 155596323 ] process 2743138, thread 1787117568, correlation 46
DRIVER cbid=677 [ 155596396 - 155596597 ] process 2743138, thread 1787117568, correlation 47
DRIVER cbid=677 [ 155596669 - 155596892 ] process 2743138, thread 1787117568, correlation 48
DRIVER cbid=677 [ 155597025 - 155597150 ] process 2743138, thread 1787117568, correlation 49
DRIVER cbid=677 [ 155597222 - 155597325 ] process 2743138, thread 1787117568, correlation 50
DRIVER cbid=677 [ 155597389 - 155597472 ] process 2743138, thread 1787117568, correlation 51
DRIVER cbid=677 [ 155597791 - 155597984 ] process 2743138, thread 1787117568, correlation 52
DRIVER cbid=677 [ 155598059 - 155598150 ] process 2743138, thread 1787117568, correlation 53
DRIVER cbid=677 [ 155598224 - 155598440 ] process 2743138, thread 1787117568, correlation 54
DRIVER cbid=677 [ 155598514 - 155598629 ] process 2743138, thread 1787117568, correlation 55
DRIVER cbid=677 [ 155598695 - 155598752 ] process 2743138, thread 1787117568, correlation 56
DRIVER cbid=677 [ 155598890 - 155599020 ] process 2743138, thread 1787117568, correlation 57
DRIVER cbid=677 [ 155599092 - 155599162 ] process 2743138, thread 1787117568, correlation 58
DRIVER cbid=677 [ 155599233 - 155599375 ] process 2743138, thread 1787117568, correlation 59
DRIVER cbid=677 [ 155599504 - 155599651 ] process 2743138, thread 1787117568, correlation 60
DRIVER cbid=677 [ 155599724 - 155599819 ] process 2743138, thread 1787117568, correlation 61
DRIVER cbid=677 [ 155599891 - 155600016 ] process 2743138, thread 1787117568, correlation 62
DRIVER cbid=677 [ 155600153 - 155600287 ] process 2743138, thread 1787117568, correlation 63
DRIVER cbid=677 [ 155600359 - 155600529 ] process 2743138, thread 1787117568, correlation 64
DRIVER cbid=677 [ 155600664 - 155600779 ] process 2743138, thread 1787117568, correlation 65
DRIVER cbid=677 [ 155600843 - 155600971 ] process 2743138, thread 1787117568, correlation 66
DRIVER cbid=677 [ 155601043 - 155601675 ] process 2743138, thread 1787117568, correlation 67
DRIVER cbid=677 [ 155601809 - 155601987 ] process 2743138, thread 1787117568, correlation 68
DRIVER cbid=677 [ 155602061 - 155602161 ] process 2743138, thread 1787117568, correlation 69
DRIVER cbid=677 [ 155602221 - 155602314 ] process 2743138, thread 1787117568, correlation 70
DRIVER cbid=677 [ 155602446 - 155602571 ] process 2743138, thread 1787117568, correlation 71
DRIVER cbid=677 [ 155602645 - 155602746 ] process 2743138, thread 1787117568, correlation 72
DRIVER cbid=677 [ 155602878 - 155602951 ] process 2743138, thread 1787117568, correlation 73
DRIVER cbid=677 [ 155603023 - 155603096 ] process 2743138, thread 1787117568, correlation 74
DRIVER cbid=677 [ 155603169 - 155603237 ] process 2743138, thread 1787117568, correlation 75
DRIVER cbid=677 [ 155603301 - 155603366 ] process 2743138, thread 1787117568, correlation 76
DRIVER cbid=677 [ 155603439 - 155603516 ] process 2743138, thread 1787117568, correlation 77
DRIVER cbid=677 [ 155603589 - 155603676 ] process 2743138, thread 1787117568, correlation 78
DRIVER cbid=677 [ 155603747 - 155603829 ] process 2743138, thread 1787117568, correlation 79
DRIVER cbid=677 [ 155603892 - 155603990 ] process 2743138, thread 1787117568, correlation 80
DRIVER cbid=677 [ 155604120 - 155604210 ] process 2743138, thread 1787117568, correlation 81
DRIVER cbid=677 [ 155604278 - 155604407 ] process 2743138, thread 1787117568, correlation 82
DRIVER cbid=677 [ 155604482 - 155604658 ] process 2743138, thread 1787117568, correlation 83
DRIVER cbid=677 [ 155604803 - 155604876 ] process 2743138, thread 1787117568, correlation 84
DRIVER cbid=677 [ 155604939 - 155605069 ] process 2743138, thread 1787117568, correlation 85
DRIVER cbid=677 [ 155605140 - 155605199 ] process 2743138, thread 1787117568, correlation 86
DRIVER cbid=677 [ 155605332 - 155605544 ] process 2743138, thread 1787117568, correlation 87
DRIVER cbid=677 [ 155605617 - 155605684 ] process 2743138, thread 1787117568, correlation 88
DRIVER cbid=677 [ 155605758 - 155605883 ] process 2743138, thread 1787117568, correlation 89
DRIVER cbid=677 [ 155605954 - 155606012 ] process 2743138, thread 1787117568, correlation 90
DRIVER cbid=677 [ 155606084 - 155606164 ] process 2743138, thread 1787117568, correlation 91
DRIVER cbid=677 [ 155606306 - 155606379 ] process 2743138, thread 1787117568, correlation 92
DRIVER cbid=677 [ 155606452 - 155606580 ] process 2743138, thread 1787117568, correlation 93
DRIVER cbid=677 [ 155606653 - 155606733 ] process 2743138, thread 1787117568, correlation 94
DRIVER cbid=677 [ 155606797 - 155606884 ] process 2743138, thread 1787117568, correlation 95
DRIVER cbid=677 [ 155606958 - 155607097 ] process 2743138, thread 1787117568, correlation 96
DRIVER cbid=677 [ 155607226 - 155607290 ] process 2743138, thread 1787117568, correlation 97
DRIVER cbid=677 [ 155607362 - 155607481 ] process 2743138, thread 1787117568, correlation 98
DRIVER cbid=677 [ 155607556 - 155607627 ] process 2743138, thread 1787117568, correlation 99
DRIVER cbid=677 [ 155607701 - 155607883 ] process 2743138, thread 1787117568, correlation 100
DRIVER cbid=677 [ 155607950 - 155608064 ] process 2743138, thread 1787117568, correlation 101
DRIVER cbid=677 [ 155608137 - 155608290 ] process 2743138, thread 1787117568, correlation 102
DRIVER cbid=677 [ 155608419 - 155608556 ] process 2743138, thread 1787117568, correlation 103
DRIVER cbid=677 [ 155608621 - 155608743 ] process 2743138, thread 1787117568, correlation 104
DRIVER cbid=677 [ 155608874 - 155608922 ] process 2743138, thread 1787117568, correlation 105
DRIVER cbid=677 [ 155608995 - 155609070 ] process 2743138, thread 1787117568, correlation 106
DRIVER cbid=677 [ 155609143 - 155609216 ] process 2743138, thread 1787117568, correlation 107
DRIVER cbid=677 [ 155609434 - 155609502 ] process 2743138, thread 1787117568, correlation 108
DRIVER cbid=677 [ 155609576 - 155609639 ] process 2743138, thread 1787117568, correlation 109
DRIVER cbid=677 [ 155609714 - 155609855 ] process 2743138, thread 1787117568, correlation 110
DRIVER cbid=677 [ 155609988 - 155610059 ] process 2743138, thread 1787117568, correlation 111
DRIVER cbid=677 [ 155610124 - 155610187 ] process 2743138, thread 1787117568, correlation 112
DRIVER cbid=677 [ 155610322 - 155610384 ] process 2743138, thread 1787117568, correlation 113
DRIVER cbid=677 [ 155610457 - 155610593 ] process 2743138, thread 1787117568, correlation 114
DRIVER cbid=677 [ 155610664 - 155610732 ] process 2743138, thread 1787117568, correlation 115
DRIVER cbid=677 [ 155610862 - 155610919 ] process 2743138, thread 1787117568, correlation 116
DRIVER cbid=677 [ 155610983 - 155611033 ] process 2743138, thread 1787117568, correlation 117
DRIVER cbid=677 [ 155611107 - 155611194 ] process 2743138, thread 1787117568, correlation 118
DRIVER cbid=677 [ 155611264 - 155611335 ] process 2743138, thread 1787117568, correlation 119
DRIVER cbid=677 [ 155611408 - 155611478 ] process 2743138, thread 1787117568, correlation 120
DRIVER cbid=677 [ 155611611 - 155611680 ] process 2743138, thread 1787117568, correlation 121
DRIVER cbid=677 [ 155611753 - 155611821 ] process 2743138, thread 1787117568, correlation 122
DRIVER cbid=677 [ 155611894 - 155611963 ] process 2743138, thread 1787117568, correlation 123
DRIVER cbid=677 [ 155612092 - 155612176 ] process 2743138, thread 1787117568, correlation 124
DRIVER cbid=677 [ 155612249 - 155612381 ] process 2743138, thread 1787117568, correlation 125
DRIVER cbid=677 [ 155612452 - 155612569 ] process 2743138, thread 1787117568, correlation 126
DRIVER cbid=677 [ 155612701 - 155612760 ] process 2743138, thread 1787117568, correlation 127
DRIVER cbid=677 [ 155612830 - 155612898 ] process 2743138, thread 1787117568, correlation 128
DRIVER cbid=677 [ 155612962 - 155613039 ] process 2743138, thread 1787117568, correlation 129
DRIVER cbid=677 [ 155613110 - 155613189 ] process 2743138, thread 1787117568, correlation 130
DRIVER cbid=677 [ 155613261 - 155613327 ] process 2743138, thread 1787117568, correlation 131
DRIVER cbid=677 [ 155613458 - 155613526 ] process 2743138, thread 1787117568, correlation 132
DRIVER cbid=677 [ 155613600 - 155613665 ] process 2743138, thread 1787117568, correlation 133
DRIVER cbid=677 [ 155613737 - 155613821 ] process 2743138, thread 1787117568, correlation 134
DRIVER cbid=677 [ 155613952 - 155614029 ] process 2743138, thread 1787117568, correlation 135
DRIVER cbid=677 [ 155614101 - 155614232 ] process 2743138, thread 1787117568, correlation 136
DRIVER cbid=677 [ 155614366 - 155614422 ] process 2743138, thread 1787117568, correlation 137
DRIVER cbid=677 [ 155614495 - 155614627 ] process 2743138, thread 1787117568, correlation 138
DRIVER cbid=677 [ 155614700 - 155614755 ] process 2743138, thread 1787117568, correlation 139
DRIVER cbid=677 [ 155614824 - 155614946 ] process 2743138, thread 1787117568, correlation 140
DRIVER cbid=677 [ 155615021 - 155615076 ] process 2743138, thread 1787117568, correlation 141
DRIVER cbid=677 [ 155615149 - 155615217 ] process 2743138, thread 1787117568, correlation 142
DRIVER cbid=677 [ 155615350 - 155615416 ] process 2743138, thread 1787117568, correlation 143
DRIVER cbid=677 [ 155615491 - 155615601 ] process 2743138, thread 1787117568, correlation 144
DRIVER cbid=677 [ 155615673 - 155615722 ] process 2743138, thread 1787117568, correlation 145
DRIVER cbid=677 [ 155615795 - 155615871 ] process 2743138, thread 1787117568, correlation 146
DRIVER cbid=677 [ 155615947 - 155616018 ] process 2743138, thread 1787117568, correlation 147
DRIVER cbid=677 [ 155616152 - 155616264 ] process 2743138, thread 1787117568, correlation 148
DRIVER cbid=677 [ 155616339 - 155616462 ] process 2743138, thread 1787117568, correlation 149
DRIVER cbid=677 [ 155616528 - 155616609 ] process 2743138, thread 1787117568, correlation 150
DRIVER cbid=677 [ 155616742 - 155616820 ] process 2743138, thread 1787117568, correlation 151
DRIVER cbid=677 [ 155616894 - 155617010 ] process 2743138, thread 1787117568, correlation 152
DRIVER cbid=677 [ 155617247 - 155617334 ] process 2743138, thread 1787117568, correlation 153
DRIVER cbid=677 [ 155617398 - 155617479 ] process 2743138, thread 1787117568, correlation 154
DRIVER cbid=677 [ 155617552 - 155617679 ] process 2743138, thread 1787117568, correlation 155
DRIVER cbid=677 [ 155617751 - 155617851 ] process 2743138, thread 1787117568, correlation 156
DRIVER cbid=677 [ 155617918 - 155618051 ] process 2743138, thread 1787117568, correlation 157
DRIVER cbid=677 [ 155618123 - 155618212 ] process 2743138, thread 1787117568, correlation 158
DRIVER cbid=677 [ 155618347 - 155618432 ] process 2743138, thread 1787117568, correlation 159
DRIVER cbid=677 [ 155618505 - 155618563 ] process 2743138, thread 1787117568, correlation 160
DRIVER cbid=677 [ 155618636 - 155618717 ] process 2743138, thread 1787117568, correlation 161
DRIVER cbid=677 [ 155618789 - 155618862 ] process 2743138, thread 1787117568, correlation 162
DRIVER cbid=677 [ 155618934 - 155619135 ] process 2743138, thread 1787117568, correlation 163
DRIVER cbid=677 [ 155619208 - 155619277 ] process 2743138, thread 1787117568, correlation 164
DRIVER cbid=677 [ 155619351 - 155619442 ] process 2743138, thread 1787117568, correlation 165
DRIVER cbid=677 [ 155619514 - 155619661 ] process 2743138, thread 1787117568, correlation 166
DRIVER cbid=677 [ 155619725 - 155619860 ] process 2743138, thread 1787117568, correlation 167
DRIVER cbid=677 [ 155619933 - 155620063 ] process 2743138, thread 1787117568, correlation 168
DRIVER cbid=677 [ 155620135 - 155620341 ] process 2743138, thread 1787117568, correlation 169
DRIVER cbid=677 [ 155620413 - 155620509 ] process 2743138, thread 1787117568, correlation 170
DRIVER cbid=677 [ 155620570 - 155620728 ] process 2743138, thread 1787117568, correlation 171
DRIVER cbid=677 [ 155620860 - 155620984 ] process 2743138, thread 1787117568, correlation 172
DRIVER cbid=677 [ 155621055 - 155621140 ] process 2743138, thread 1787117568, correlation 173
DRIVER cbid=677 [ 155621213 - 155621324 ] process 2743138, thread 1787117568, correlation 174
DRIVER cbid=677 [ 155621458 - 155621544 ] process 2743138, thread 1787117568, correlation 175
DRIVER cbid=677 [ 155621615 - 155621697 ] process 2743138, thread 1787117568, correlation 176
DRIVER cbid=677 [ 155621768 - 155621914 ] process 2743138, thread 1787117568, correlation 177
DRIVER cbid=677 [ 155621984 - 155622094 ] process 2743138, thread 1787117568, correlation 178
DRIVER cbid=677 [ 155622165 - 155622257 ] process 2743138, thread 1787117568, correlation 179
DRIVER cbid=677 [ 155622321 - 155622496 ] process 2743138, thread 1787117568, correlation 180
DRIVER cbid=677 [ 155622570 - 155622665 ] process 2743138, thread 1787117568, correlation 181
DRIVER cbid=677 [ 155622741 - 155622897 ] process 2743138, thread 1787117568, correlation 182
DRIVER cbid=677 [ 155623030 - 155623123 ] process 2743138, thread 1787117568, correlation 183
DRIVER cbid=677 [ 155623197 - 155623321 ] process 2743138, thread 1787117568, correlation 184
DRIVER cbid=677 [ 155623385 - 155623527 ] process 2743138, thread 1787117568, correlation 185
DRIVER cbid=677 [ 155623596 - 155623659 ] process 2743138, thread 1787117568, correlation 186
DRIVER cbid=677 [ 155623731 - 155623790 ] process 2743138, thread 1787117568, correlation 187
DRIVER cbid=677 [ 155623924 - 155624012 ] process 2743138, thread 1787117568, correlation 188
DRIVER cbid=677 [ 155624086 - 155624171 ] process 2743138, thread 1787117568, correlation 189
DRIVER cbid=677 [ 155624245 - 155624398 ] process 2743138, thread 1787117568, correlation 190
DRIVER cbid=677 [ 155624531 - 155624599 ] process 2743138, thread 1787117568, correlation 191
DRIVER cbid=677 [ 155624671 - 155624738 ] process 2743138, thread 1787117568, correlation 192
DRIVER cbid=677 [ 155625055 - 155625181 ] process 2743138, thread 1787117568, correlation 193
DRIVER cbid=677 [ 155625252 - 155625319 ] process 2743138, thread 1787117568, correlation 194
DRIVER cbid=677 [ 155625381 - 155625463 ] process 2743138, thread 1787117568, correlation 195
DRIVER cbid=677 [ 155625594 - 155625689 ] process 2743138, thread 1787117568, correlation 196
DRIVER cbid=677 [ 155625762 - 155625829 ] process 2743138, thread 1787117568, correlation 197
DRIVER cbid=677 [ 155625899 - 155625977 ] process 2743138, thread 1787117568, correlation 198
DRIVER cbid=677 [ 155626109 - 155626203 ] process 2743138, thread 1787117568, correlation 199
DRIVER cbid=677 [ 155626275 - 155626340 ] process 2743138, thread 1787117568, correlation 200
DRIVER cbid=677 [ 155626473 - 155626554 ] process 2743138, thread 1787117568, correlation 201
DRIVER cbid=677 [ 155626621 - 155626714 ] process 2743138, thread 1787117568, correlation 202
DRIVER cbid=677 [ 155626789 - 155626862 ] process 2743138, thread 1787117568, correlation 203
DRIVER cbid=677 [ 155626997 - 155627059 ] process 2743138, thread 1787117568, correlation 204
DRIVER cbid=677 [ 155627131 - 155627203 ] process 2743138, thread 1787117568, correlation 205
DRIVER cbid=677 [ 155627276 - 155627340 ] process 2743138, thread 1787117568, correlation 206
DRIVER cbid=677 [ 155627412 - 155627504 ] process 2743138, thread 1787117568, correlation 207
DRIVER cbid=677 [ 155627579 - 155627654 ] process 2743138, thread 1787117568, correlation 208
DRIVER cbid=677 [ 155627789 - 155627885 ] process 2743138, thread 1787117568, correlation 209
DRIVER cbid=677 [ 155627958 - 155628031 ] process 2743138, thread 1787117568, correlation 210
DRIVER cbid=677 [ 155628104 - 155628162 ] process 2743138, thread 1787117568, correlation 211
DRIVER cbid=677 [ 155628296 - 155628361 ] process 2743138, thread 1787117568, correlation 212
DRIVER cbid=677 [ 155628434 - 155628489 ] process 2743138, thread 1787117568, correlation 213
DRIVER cbid=677 [ 155628561 - 155628614 ] process 2743138, thread 1787117568, correlation 214
DRIVER cbid=677 [ 155628964 - 155629026 ] process 2743138, thread 1787117568, correlation 215
DRIVER cbid=677 [ 155629100 - 155629183 ] process 2743138, thread 1787117568, correlation 216
DRIVER cbid=677 [ 155629317 - 155629437 ] process 2743138, thread 1787117568, correlation 217
DRIVER cbid=677 [ 155629514 - 155629577 ] process 2743138, thread 1787117568, correlation 218
DRIVER cbid=677 [ 155629648 - 155629721 ] process 2743138, thread 1787117568, correlation 219
DRIVER cbid=677 [ 155629793 - 155629878 ] process 2743138, thread 1787117568, correlation 220
DRIVER cbid=677 [ 155629951 - 155630019 ] process 2743138, thread 1787117568, correlation 221
DRIVER cbid=677 [ 155630090 - 155630154 ] process 2743138, thread 1787117568, correlation 222
DRIVER cbid=677 [ 155630226 - 155630308 ] process 2743138, thread 1787117568, correlation 223
DRIVER cbid=677 [ 155630381 - 155630458 ] process 2743138, thread 1787117568, correlation 224
DRIVER cbid=677 [ 155630530 - 155630606 ] process 2743138, thread 1787117568, correlation 225
DRIVER cbid=677 [ 155630679 - 155630771 ] process 2743138, thread 1787117568, correlation 226
DRIVER cbid=677 [ 155630843 - 155630915 ] process 2743138, thread 1787117568, correlation 227
DRIVER cbid=677 [ 155631047 - 155631123 ] process 2743138, thread 1787117568, correlation 228
DRIVER cbid=677 [ 155631195 - 155631288 ] process 2743138, thread 1787117568, correlation 229
DRIVER cbid=677 [ 155631358 - 155631431 ] process 2743138, thread 1787117568, correlation 230
DRIVER cbid=677 [ 155631560 - 155631637 ] process 2743138, thread 1787117568, correlation 231
DRIVER cbid=677 [ 155631711 - 155631785 ] process 2743138, thread 1787117568, correlation 232
DRIVER cbid=677 [ 155631916 - 155631982 ] process 2743138, thread 1787117568, correlation 233
DRIVER cbid=677 [ 155632053 - 155632117 ] process 2743138, thread 1787117568, correlation 234
DRIVER cbid=677 [ 155632188 - 155632255 ] process 2743138, thread 1787117568, correlation 235
DRIVER cbid=677 [ 155632384 - 155632442 ] process 2743138, thread 1787117568, correlation 236
DRIVER cbid=677 [ 155632513 - 155632599 ] process 2743138, thread 1787117568, correlation 237
DRIVER cbid=677 [ 155632670 - 155632755 ] process 2743138, thread 1787117568, correlation 238
DRIVER cbid=677 [ 155632884 - 155632991 ] process 2743138, thread 1787117568, correlation 239
DRIVER cbid=677 [ 155633063 - 155633138 ] process 2743138, thread 1787117568, correlation 240
DRIVER cbid=677 [ 155633212 - 155633289 ] process 2743138, thread 1787117568, correlation 241
DRIVER cbid=677 [ 155633365 - 155633439 ] process 2743138, thread 1787117568, correlation 242
DRIVER cbid=677 [ 155633513 - 155633584 ] process 2743138, thread 1787117568, correlation 243
DRIVER cbid=677 [ 155633716 - 155633827 ] process 2743138, thread 1787117568, correlation 244
DRIVER cbid=677 [ 155633904 - 155633981 ] process 2743138, thread 1787117568, correlation 245
DRIVER cbid=677 [ 155634054 - 155634192 ] process 2743138, thread 1787117568, correlation 246
DRIVER cbid=677 [ 155634324 - 155634398 ] process 2743138, thread 1787117568, correlation 247
DRIVER cbid=677 [ 155634471 - 155634533 ] process 2743138, thread 1787117568, correlation 248
DRIVER cbid=677 [ 155634605 - 155634677 ] process 2743138, thread 1787117568, correlation 249
DRIVER cbid=677 [ 155634743 - 155634807 ] process 2743138, thread 1787117568, correlation 250
DRIVER cbid=677 [ 155634880 - 155634966 ] process 2743138, thread 1787117568, correlation 251
DRIVER cbid=677 [ 155635029 - 155635158 ] process 2743138, thread 1787117568, correlation 252
DRIVER cbid=677 [ 155635225 - 155635293 ] process 2743138, thread 1787117568, correlation 253
DRIVER cbid=677 [ 155635364 - 155635447 ] process 2743138, thread 1787117568, correlation 254
DRIVER cbid=677 [ 155635576 - 155635671 ] process 2743138, thread 1787117568, correlation 255
DRIVER cbid=677 [ 155635736 - 155635811 ] process 2743138, thread 1787117568, correlation 256
DRIVER cbid=677 [ 155635942 - 155636014 ] process 2743138, thread 1787117568, correlation 257
DRIVER cbid=677 [ 155636086 - 155636180 ] process 2743138, thread 1787117568, correlation 258
DRIVER cbid=677 [ 155636252 - 155636344 ] process 2743138, thread 1787117568, correlation 259
DRIVER cbid=677 [ 155636773 - 155636840 ] process 2743138, thread 1787117568, correlation 260
DRIVER cbid=677 [ 155636905 - 155637000 ] process 2743138, thread 1787117568, correlation 261
DRIVER cbid=677 [ 155637074 - 155637163 ] process 2743138, thread 1787117568, correlation 262
DRIVER cbid=677 [ 155637305 - 155637376 ] process 2743138, thread 1787117568, correlation 263
DRIVER cbid=677 [ 155637446 - 155637543 ] process 2743138, thread 1787117568, correlation 264
DRIVER cbid=677 [ 155637615 - 155637707 ] process 2743138, thread 1787117568, correlation 265
DRIVER cbid=677 [ 155637780 - 155637864 ] process 2743138, thread 1787117568, correlation 266
DRIVER cbid=677 [ 155637937 - 155638075 ] process 2743138, thread 1787117568, correlation 267
DRIVER cbid=677 [ 155638208 - 155638355 ] process 2743138, thread 1787117568, correlation 268
DRIVER cbid=677 [ 155638426 - 155638584 ] process 2743138, thread 1787117568, correlation 269
DRIVER cbid=677 [ 155638657 - 155638755 ] process 2743138, thread 1787117568, correlation 270
DRIVER cbid=677 [ 155638889 - 155638989 ] process 2743138, thread 1787117568, correlation 271
DRIVER cbid=677 [ 155639057 - 155639156 ] process 2743138, thread 1787117568, correlation 272
DRIVER cbid=677 [ 155639289 - 155639388 ] process 2743138, thread 1787117568, correlation 273
DRIVER cbid=677 [ 155639453 - 155639549 ] process 2743138, thread 1787117568, correlation 274
DRIVER cbid=677 [ 155639623 - 155639765 ] process 2743138, thread 1787117568, correlation 275
DRIVER cbid=677 [ 155639900 - 155640016 ] process 2743138, thread 1787117568, correlation 276
DRIVER cbid=677 [ 155640090 - 155640183 ] process 2743138, thread 1787117568, correlation 277
DRIVER cbid=677 [ 155640255 - 155640353 ] process 2743138, thread 1787117568, correlation 278
DRIVER cbid=677 [ 155640424 - 155640501 ] process 2743138, thread 1787117568, correlation 279
DRIVER cbid=677 [ 155640574 - 155640657 ] process 2743138, thread 1787117568, correlation 280
DRIVER cbid=677 [ 155640789 - 155640872 ] process 2743138, thread 1787117568, correlation 281
DRIVER cbid=677 [ 155640949 - 155641059 ] process 2743138, thread 1787117568, correlation 282
DRIVER cbid=677 [ 155641134 - 155641206 ] process 2743138, thread 1787117568, correlation 283
DRIVER cbid=677 [ 155641280 - 155641386 ] process 2743138, thread 1787117568, correlation 284
DRIVER cbid=677 [ 155641460 - 155641603 ] process 2743138, thread 1787117568, correlation 285
DRIVER cbid=677 [ 155641678 - 155641742 ] process 2743138, thread 1787117568, correlation 286
DRIVER cbid=677 [ 155641876 - 155641942 ] process 2743138, thread 1787117568, correlation 287
DRIVER cbid=677 [ 155642013 - 155642077 ] process 2743138, thread 1787117568, correlation 288
DRIVER cbid=677 [ 155642149 - 155642220 ] process 2743138, thread 1787117568, correlation 289
DRIVER cbid=677 [ 155642292 - 155642366 ] process 2743138, thread 1787117568, correlation 290
DRIVER cbid=677 [ 155642438 - 155642511 ] process 2743138, thread 1787117568, correlation 291
DRIVER cbid=677 [ 155642585 - 155642706 ] process 2743138, thread 1787117568, correlation 292
DRIVER cbid=677 [ 155642785 - 155642859 ] process 2743138, thread 1787117568, correlation 293
DRIVER cbid=677 [ 155642933 - 155643123 ] process 2743138, thread 1787117568, correlation 294
DRIVER cbid=677 [ 155643190 - 155643365 ] process 2743138, thread 1787117568, correlation 295
DRIVER cbid=677 [ 155643439 - 155643578 ] process 2743138, thread 1787117568, correlation 296
DRIVER cbid=677 [ 155643711 - 155643864 ] process 2743138, thread 1787117568, correlation 297
DRIVER cbid=677 [ 155643937 - 155644029 ] process 2743138, thread 1787117568, correlation 298
DRIVER cbid=677 [ 155644100 - 155644203 ] process 2743138, thread 1787117568, correlation 299
DRIVER cbid=677 [ 155644271 - 155644367 ] process 2743138, thread 1787117568, correlation 300
DRIVER cbid=677 [ 155644440 - 155644510 ] process 2743138, thread 1787117568, correlation 301
DRIVER cbid=677 [ 155644581 - 155644664 ] process 2743138, thread 1787117568, correlation 302
DRIVER cbid=677 [ 155644805 - 155644876 ] process 2743138, thread 1787117568, correlation 303
DRIVER cbid=677 [ 155644949 - 155645058 ] process 2743138, thread 1787117568, correlation 304
DRIVER cbid=677 [ 155645187 - 155645301 ] process 2743138, thread 1787117568, correlation 305
DRIVER cbid=677 [ 155645369 - 155645488 ] process 2743138, thread 1787117568, correlation 306
DRIVER cbid=677 [ 155645561 - 155645636 ] process 2743138, thread 1787117568, correlation 307
DRIVER cbid=677 [ 155645777 - 155645853 ] process 2743138, thread 1787117568, correlation 308
DRIVER cbid=677 [ 155645925 - 155646000 ] process 2743138, thread 1787117568, correlation 309
DRIVER cbid=677 [ 155646073 - 155646163 ] process 2743138, thread 1787117568, correlation 310
DRIVER cbid=677 [ 155646295 - 155646393 ] process 2743138, thread 1787117568, correlation 311
DRIVER cbid=677 [ 155646464 - 155646554 ] process 2743138, thread 1787117568, correlation 312
DRIVER cbid=677 [ 155646684 - 155646771 ] process 2743138, thread 1787117568, correlation 313
DRIVER cbid=677 [ 155646843 - 155646913 ] process 2743138, thread 1787117568, correlation 314
DRIVER cbid=677 [ 155646985 - 155647060 ] process 2743138, thread 1787117568, correlation 315
DRIVER cbid=677 [ 155647130 - 155647230 ] process 2743138, thread 1787117568, correlation 316
DRIVER cbid=677 [ 155647301 - 155647434 ] process 2743138, thread 1787117568, correlation 317
DRIVER cbid=677 [ 155647504 - 155647580 ] process 2743138, thread 1787117568, correlation 318
DRIVER cbid=677 [ 155647644 - 155647783 ] process 2743138, thread 1787117568, correlation 319
DRIVER cbid=677 [ 155647855 - 155647940 ] process 2743138, thread 1787117568, correlation 320
DRIVER cbid=677 [ 155648071 - 155648141 ] process 2743138, thread 1787117568, correlation 321
DRIVER cbid=677 [ 155648214 - 155648285 ] process 2743138, thread 1787117568, correlation 322
DRIVER cbid=677 [ 155648357 - 155648476 ] process 2743138, thread 1787117568, correlation 323
DRIVER cbid=677 [ 155648547 - 155648644 ] process 2743138, thread 1787117568, correlation 324
DRIVER cbid=677 [ 155648706 - 155648781 ] process 2743138, thread 1787117568, correlation 325
DRIVER cbid=677 [ 155648856 - 155648929 ] process 2743138, thread 1787117568, correlation 326
DRIVER cbid=677 [ 155649062 - 155649146 ] process 2743138, thread 1787117568, correlation 327
DRIVER cbid=677 [ 155649218 - 155649309 ] process 2743138, thread 1787117568, correlation 328
DRIVER cbid=677 [ 155649381 - 155649453 ] process 2743138, thread 1787117568, correlation 329
DRIVER cbid=677 [ 155649527 - 155649692 ] process 2743138, thread 1787117568, correlation 330
DRIVER cbid=677 [ 155649766 - 155649982 ] process 2743138, thread 1787117568, correlation 331
DRIVER cbid=677 [ 155650111 - 155650209 ] process 2743138, thread 1787117568, correlation 332
DRIVER cbid=677 [ 155650282 - 155650395 ] process 2743138, thread 1787117568, correlation 333
DRIVER cbid=677 [ 155650469 - 155650609 ] process 2743138, thread 1787117568, correlation 334
DRIVER cbid=677 [ 155650733 - 155650854 ] process 2743138, thread 1787117568, correlation 335
DRIVER cbid=677 [ 155650927 - 155651066 ] process 2743138, thread 1787117568, correlation 336
DRIVER cbid=677 [ 155651196 - 155651329 ] process 2743138, thread 1787117568, correlation 337
DRIVER cbid=677 [ 155651402 - 155651495 ] process 2743138, thread 1787117568, correlation 338
DRIVER cbid=677 [ 155651567 - 155651716 ] process 2743138, thread 1787117568, correlation 339
DRIVER cbid=677 [ 155651840 - 155651920 ] process 2743138, thread 1787117568, correlation 340
DRIVER cbid=677 [ 155651992 - 155652388 ] process 2743138, thread 1787117568, correlation 341
DRIVER cbid=677 [ 155652452 - 155652531 ] process 2743138, thread 1787117568, correlation 342
DRIVER cbid=677 [ 155652664 - 155652790 ] process 2743138, thread 1787117568, correlation 343
DRIVER cbid=677 [ 155652864 - 155652961 ] process 2743138, thread 1787117568, correlation 344
DRIVER cbid=677 [ 155653033 - 155653084 ] process 2743138, thread 1787117568, correlation 345
DRIVER cbid=677 [ 155653158 - 155653227 ] process 2743138, thread 1787117568, correlation 346
DRIVER cbid=677 [ 155653299 - 155653371 ] process 2743138, thread 1787117568, correlation 347
DRIVER cbid=677 [ 155653503 - 155653565 ] process 2743138, thread 1787117568, correlation 348
DRIVER cbid=677 [ 155653636 - 155653770 ] process 2743138, thread 1787117568, correlation 349
DRIVER cbid=677 [ 155653841 - 155653915 ] process 2743138, thread 1787117568, correlation 350
DRIVER cbid=677 [ 155654046 - 155654106 ] process 2743138, thread 1787117568, correlation 351
DRIVER cbid=677 [ 155654171 - 155654273 ] process 2743138, thread 1787117568, correlation 352
DRIVER cbid=677 [ 155654403 - 155654490 ] process 2743138, thread 1787117568, correlation 353
DRIVER cbid=677 [ 155654553 - 155654636 ] process 2743138, thread 1787117568, correlation 354
DRIVER cbid=677 [ 155654706 - 155654778 ] process 2743138, thread 1787117568, correlation 355
DRIVER cbid=677 [ 155654905 - 155654983 ] process 2743138, thread 1787117568, correlation 356
DRIVER cbid=677 [ 155655054 - 155655119 ] process 2743138, thread 1787117568, correlation 357
DRIVER cbid=677 [ 155655191 - 155655291 ] process 2743138, thread 1787117568, correlation 358
DRIVER cbid=677 [ 155655412 - 155655513 ] process 2743138, thread 1787117568, correlation 359
DRIVER cbid=677 [ 155655583 - 155655641 ] process 2743138, thread 1787117568, correlation 360
DRIVER cbid=677 [ 155655713 - 155655784 ] process 2743138, thread 1787117568, correlation 361
DRIVER cbid=677 [ 155655858 - 155655921 ] process 2743138, thread 1787117568, correlation 362
DRIVER cbid=677 [ 155655994 - 155656087 ] process 2743138, thread 1787117568, correlation 363
DRIVER cbid=677 [ 155656220 - 155656286 ] process 2743138, thread 1787117568, correlation 364
DRIVER cbid=677 [ 155656360 - 155656423 ] process 2743138, thread 1787117568, correlation 365
DRIVER cbid=677 [ 155656496 - 155656554 ] process 2743138, thread 1787117568, correlation 366
DRIVER cbid=677 [ 155656686 - 155656747 ] process 2743138, thread 1787117568, correlation 367
DRIVER cbid=677 [ 155656821 - 155656877 ] process 2743138, thread 1787117568, correlation 368
DRIVER cbid=677 [ 155657006 - 155657082 ] process 2743138, thread 1787117568, correlation 369
DRIVER cbid=677 [ 155657154 - 155657230 ] process 2743138, thread 1787117568, correlation 370
DRIVER cbid=677 [ 155657301 - 155657371 ] process 2743138, thread 1787117568, correlation 371
DRIVER cbid=677 [ 155657501 - 155657574 ] process 2743138, thread 1787117568, correlation 372
DRIVER cbid=677 [ 155657645 - 155657710 ] process 2743138, thread 1787117568, correlation 373
DRIVER cbid=677 [ 155657781 - 155657850 ] process 2743138, thread 1787117568, correlation 374
DRIVER cbid=677 [ 155657981 - 155658050 ] process 2743138, thread 1787117568, correlation 375
DRIVER cbid=677 [ 155658121 - 155658206 ] process 2743138, thread 1787117568, correlation 376
DRIVER cbid=677 [ 155658336 - 155658415 ] process 2743138, thread 1787117568, correlation 377
DRIVER cbid=677 [ 155658488 - 155658599 ] process 2743138, thread 1787117568, correlation 378
DRIVER cbid=677 [ 155658671 - 155658772 ] process 2743138, thread 1787117568, correlation 379
DRIVER cbid=677 [ 155658845 - 155658943 ] process 2743138, thread 1787117568, correlation 380
DRIVER cbid=677 [ 155659014 - 155659101 ] process 2743138, thread 1787117568, correlation 381
DRIVER cbid=677 [ 155659173 - 155659253 ] process 2743138, thread 1787117568, correlation 382
DRIVER cbid=677 [ 155659383 - 155659455 ] process 2743138, thread 1787117568, correlation 383
DRIVER cbid=677 [ 155659518 - 155659598 ] process 2743138, thread 1787117568, correlation 384
DRIVER cbid=677 [ 155659726 - 155659799 ] process 2743138, thread 1787117568, correlation 385
DRIVER cbid=677 [ 155659872 - 155660316 ] process 2743138, thread 1787117568, correlation 386
DRIVER cbid=677 [ 155660379 - 155660471 ] process 2743138, thread 1787117568, correlation 387
DRIVER cbid=677 [ 155660601 - 155660693 ] process 2743138, thread 1787117568, correlation 388
DRIVER cbid=677 [ 155660768 - 155660846 ] process 2743138, thread 1787117568, correlation 389
DRIVER cbid=677 [ 155660919 - 155661040 ] process 2743138, thread 1787117568, correlation 390
DRIVER cbid=677 [ 155661104 - 155661274 ] process 2743138, thread 1787117568, correlation 391
DRIVER cbid=677 [ 155661352 - 155661461 ] process 2743138, thread 1787117568, correlation 392
DRIVER cbid=677 [ 155661598 - 155661690 ] process 2743138, thread 1787117568, correlation 393
DRIVER cbid=677 [ 155661755 - 155661873 ] process 2743138, thread 1787117568, correlation 394
DRIVER cbid=677 [ 155661946 - 155662065 ] process 2743138, thread 1787117568, correlation 395
DRIVER cbid=677 [ 155662136 - 155662212 ] process 2743138, thread 1787117568, correlation 396
DRIVER cbid=677 [ 155662274 - 155662353 ] process 2743138, thread 1787117568, correlation 397
DRIVER cbid=677 [ 155662424 - 155662518 ] process 2743138, thread 1787117568, correlation 398
DRIVER cbid=677 [ 155662650 - 155662743 ] process 2743138, thread 1787117568, correlation 399
DRIVER cbid=677 [ 155662815 - 155662948 ] process 2743138, thread 1787117568, correlation 400
DRIVER cbid=677 [ 155663082 - 155663191 ] process 2743138, thread 1787117568, correlation 401
DRIVER cbid=677 [ 155663263 - 155663337 ] process 2743138, thread 1787117568, correlation 402
DRIVER cbid=677 [ 155663407 - 155663486 ] process 2743138, thread 1787117568, correlation 403
DRIVER cbid=677 [ 155663569 - 155663643 ] process 2743138, thread 1787117568, correlation 404
DRIVER cbid=677 [ 155663715 - 155663779 ] process 2743138, thread 1787117568, correlation 405
DRIVER cbid=677 [ 155663848 - 155663928 ] process 2743138, thread 1787117568, correlation 406
DRIVER cbid=677 [ 155664056 - 155664139 ] process 2743138, thread 1787117568, correlation 407
DRIVER cbid=677 [ 155664211 - 155664317 ] process 2743138, thread 1787117568, correlation 408
DRIVER cbid=677 [ 155664452 - 155664529 ] process 2743138, thread 1787117568, correlation 409
DRIVER cbid=677 [ 155664603 - 155664674 ] process 2743138, thread 1787117568, correlation 410
DRIVER cbid=677 [ 155664748 - 155664821 ] process 2743138, thread 1787117568, correlation 411
DRIVER cbid=677 [ 155664895 - 155664951 ] process 2743138, thread 1787117568, correlation 412
DRIVER cbid=677 [ 155665018 - 155665077 ] process 2743138, thread 1787117568, correlation 413
DRIVER cbid=677 [ 155665148 - 155665198 ] process 2743138, thread 1787117568, correlation 414
DRIVER cbid=677 [ 155665270 - 155665353 ] process 2743138, thread 1787117568, correlation 415
DRIVER cbid=677 [ 155665425 - 155665521 ] process 2743138, thread 1787117568, correlation 416
DRIVER cbid=677 [ 155665593 - 155665685 ] process 2743138, thread 1787117568, correlation 417
DRIVER cbid=677 [ 155665751 - 155665850 ] process 2743138, thread 1787117568, correlation 418
DRIVER cbid=677 [ 155665923 - 155666028 ] process 2743138, thread 1787117568, correlation 419
DRIVER cbid=2 [ 155666461 - 155666525 ] process 2743138, thread 1787117568, correlation 420
DRIVER cbid=1 [ 155666765 - 155667424 ] process 2743138, thread 1787117568, correlation 421
DRIVER cbid=673 [ 155668473 - 155668642 ] process 2743138, thread 1787117568, correlation 422
DRIVER cbid=4 [ 155677560 - 155677611 ] process 2743138, thread 1787117568, correlation 423
DRIVER cbid=3 [ 155677722 - 155677778 ] process 2743138, thread 1787117568, correlation 424
DRIVER cbid=5 [ 155680119 - 155683385 ] process 2743138, thread 1787117568, correlation 425
DRIVER cbid=259 [ 155683596 - 155686445 ] process 2743138, thread 1787117568, correlation 426
DRIVER cbid=9 [ 155686620 - 155686878 ] process 2743138, thread 1787117568, correlation 427
DRIVER cbid=9 [ 155686964 - 155687018 ] process 2743138, thread 1787117568, correlation 428
DRIVER cbid=9 [ 155687087 - 155687147 ] process 2743138, thread 1787117568, correlation 429
DRIVER cbid=9 [ 155687215 - 155687282 ] process 2743138, thread 1787117568, correlation 430
DRIVER cbid=9 [ 155687350 - 155687426 ] process 2743138, thread 1787117568, correlation 431
DRIVER cbid=9 [ 155687491 - 155692357 ] process 2743138, thread 1787117568, correlation 432
DRIVER cbid=9 [ 155692479 - 155692540 ] process 2743138, thread 1787117568, correlation 433
DRIVER cbid=9 [ 155692618 - 155692664 ] process 2743138, thread 1787117568, correlation 434
DRIVER cbid=9 [ 155692738 - 155692789 ] process 2743138, thread 1787117568, correlation 435
DRIVER cbid=9 [ 155692861 - 155692911 ] process 2743138, thread 1787117568, correlation 436
DRIVER cbid=9 [ 155692981 - 155693121 ] process 2743138, thread 1787117568, correlation 437
DRIVER cbid=9 [ 155693204 - 155693251 ] process 2743138, thread 1787117568, correlation 438
DRIVER cbid=9 [ 155693317 - 155693377 ] process 2743138, thread 1787117568, correlation 439
DRIVER cbid=9 [ 155693443 - 155693492 ] process 2743138, thread 1787117568, correlation 440
DRIVER cbid=9 [ 155693558 - 155693605 ] process 2743138, thread 1787117568, correlation 441
DRIVER cbid=9 [ 155693671 - 155693719 ] process 2743138, thread 1787117568, correlation 442
DRIVER cbid=9 [ 155693785 - 155693833 ] process 2743138, thread 1787117568, correlation 443
DRIVER cbid=9 [ 155693898 - 155693945 ] process 2743138, thread 1787117568, correlation 444
DRIVER cbid=9 [ 155694012 - 155694070 ] process 2743138, thread 1787117568, correlation 445
DRIVER cbid=9 [ 155694136 - 155694184 ] process 2743138, thread 1787117568, correlation 446
DRIVER cbid=9 [ 155694249 - 155694297 ] process 2743138, thread 1787117568, correlation 447
DRIVER cbid=9 [ 155694363 - 155694411 ] process 2743138, thread 1787117568, correlation 448
DRIVER cbid=9 [ 155694476 - 155694523 ] process 2743138, thread 1787117568, correlation 449
DRIVER cbid=9 [ 155694592 - 155694639 ] process 2743138, thread 1787117568, correlation 450
DRIVER cbid=9 [ 155694705 - 155694753 ] process 2743138, thread 1787117568, correlation 451
DRIVER cbid=9 [ 155694817 - 155694875 ] process 2743138, thread 1787117568, correlation 452
DRIVER cbid=9 [ 155694941 - 155694988 ] process 2743138, thread 1787117568, correlation 453
DRIVER cbid=9 [ 155695053 - 155695106 ] process 2743138, thread 1787117568, correlation 454
DRIVER cbid=9 [ 155695172 - 155695226 ] process 2743138, thread 1787117568, correlation 455
DRIVER cbid=9 [ 155695291 - 155695348 ] process 2743138, thread 1787117568, correlation 456
DRIVER cbid=9 [ 155695412 - 155695467 ] process 2743138, thread 1787117568, correlation 457
DRIVER cbid=9 [ 155695533 - 155695588 ] process 2743138, thread 1787117568, correlation 458
DRIVER cbid=9 [ 155695653 - 155695718 ] process 2743138, thread 1787117568, correlation 459
DRIVER cbid=9 [ 155695784 - 155695838 ] process 2743138, thread 1787117568, correlation 460
DRIVER cbid=9 [ 155695902 - 155695957 ] process 2743138, thread 1787117568, correlation 461
DRIVER cbid=9 [ 155696022 - 155696076 ] process 2743138, thread 1787117568, correlation 462
DRIVER cbid=9 [ 155696141 - 155696197 ] process 2743138, thread 1787117568, correlation 463
DRIVER cbid=9 [ 155696263 - 155696317 ] process 2743138, thread 1787117568, correlation 464
DRIVER cbid=9 [ 155696375 - 155696430 ] process 2743138, thread 1787117568, correlation 465
DRIVER cbid=9 [ 155696488 - 155696543 ] process 2743138, thread 1787117568, correlation 466
DRIVER cbid=9 [ 155696601 - 155696656 ] process 2743138, thread 1787117568, correlation 467
DRIVER cbid=9 [ 155696713 - 155696769 ] process 2743138, thread 1787117568, correlation 468
DRIVER cbid=9 [ 155696826 - 155696881 ] process 2743138, thread 1787117568, correlation 469
DRIVER cbid=9 [ 155696938 - 155696994 ] process 2743138, thread 1787117568, correlation 470
DRIVER cbid=9 [ 155697052 - 155697107 ] process 2743138, thread 1787117568, correlation 471
DRIVER cbid=9 [ 155697166 - 155697221 ] process 2743138, thread 1787117568, correlation 472
DRIVER cbid=9 [ 155697279 - 155697334 ] process 2743138, thread 1787117568, correlation 473
DRIVER cbid=9 [ 155697392 - 155697447 ] process 2743138, thread 1787117568, correlation 474
DRIVER cbid=9 [ 155697512 - 155697562 ] process 2743138, thread 1787117568, correlation 475
DRIVER cbid=9 [ 155697620 - 155697676 ] process 2743138, thread 1787117568, correlation 476
DRIVER cbid=9 [ 155697733 - 155697788 ] process 2743138, thread 1787117568, correlation 477
DRIVER cbid=9 [ 155697847 - 155697902 ] process 2743138, thread 1787117568, correlation 478
DRIVER cbid=9 [ 155697962 - 155698016 ] process 2743138, thread 1787117568, correlation 479
DRIVER cbid=9 [ 155698076 - 155698130 ] process 2743138, thread 1787117568, correlation 480
DRIVER cbid=9 [ 155698189 - 155698244 ] process 2743138, thread 1787117568, correlation 481
DRIVER cbid=9 [ 155698304 - 157372055 ] process 2743138, thread 1787117568, correlation 482
DRIVER cbid=9 [ 157372231 - 157372305 ] process 2743138, thread 1787117568, correlation 483
DRIVER cbid=9 [ 157372377 - 157372437 ] process 2743138, thread 1787117568, correlation 484
DRIVER cbid=9 [ 157372504 - 157372563 ] process 2743138, thread 1787117568, correlation 485
DRIVER cbid=9 [ 157372629 - 157372686 ] process 2743138, thread 1787117568, correlation 486
DRIVER cbid=9 [ 157372749 - 157372807 ] process 2743138, thread 1787117568, correlation 487
DRIVER cbid=9 [ 157372868 - 157372924 ] process 2743138, thread 1787117568, correlation 488
DRIVER cbid=9 [ 157372984 - 157373053 ] process 2743138, thread 1787117568, correlation 489
DRIVER cbid=9 [ 157373119 - 157373177 ] process 2743138, thread 1787117568, correlation 490
DRIVER cbid=9 [ 157373235 - 157373292 ] process 2743138, thread 1787117568, correlation 491
DRIVER cbid=9 [ 157373351 - 157373405 ] process 2743138, thread 1787117568, correlation 492
DRIVER cbid=9 [ 157373465 - 157373522 ] process 2743138, thread 1787117568, correlation 493
DRIVER cbid=9 [ 157373582 - 157373638 ] process 2743138, thread 1787117568, correlation 494
DRIVER cbid=9 [ 157373698 - 157373754 ] process 2743138, thread 1787117568, correlation 495
DRIVER cbid=9 [ 157373814 - 157373927 ] process 2743138, thread 1787117568, correlation 496
DRIVER cbid=9 [ 157373987 - 157374042 ] process 2743138, thread 1787117568, correlation 497
DRIVER cbid=9 [ 157374102 - 157374150 ] process 2743138, thread 1787117568, correlation 498
DRIVER cbid=9 [ 157374208 - 157374264 ] process 2743138, thread 1787117568, correlation 499
DRIVER cbid=9 [ 157374324 - 157374391 ] process 2743138, thread 1787117568, correlation 500
DRIVER cbid=9 [ 157374451 - 157374508 ] process 2743138, thread 1787117568, correlation 501
DRIVER cbid=9 [ 157374567 - 157374623 ] process 2743138, thread 1787117568, correlation 502
DRIVER cbid=9 [ 157374683 - 157374738 ] process 2743138, thread 1787117568, correlation 503
DRIVER cbid=9 [ 157374797 - 158730479 ] process 2743138, thread 1787117568, correlation 504
DRIVER cbid=9 [ 158730621 - 158730700 ] process 2743138, thread 1787117568, correlation 505
DRIVER cbid=9 [ 158730770 - 158730828 ] process 2743138, thread 1787117568, correlation 506
DRIVER cbid=9 [ 158730894 - 158730952 ] process 2743138, thread 1787117568, correlation 507
DRIVER cbid=9 [ 158731018 - 158731073 ] process 2743138, thread 1787117568, correlation 508
DRIVER cbid=9 [ 158731133 - 158731192 ] process 2743138, thread 1787117568, correlation 509
DRIVER cbid=9 [ 158731258 - 158731315 ] process 2743138, thread 1787117568, correlation 510
DRIVER cbid=9 [ 158731375 - 158731432 ] process 2743138, thread 1787117568, correlation 511
DRIVER cbid=9 [ 158731492 - 158731968 ] process 2743138, thread 1787117568, correlation 512
DRIVER cbid=9 [ 158732034 - 158732105 ] process 2743138, thread 1787117568, correlation 513
DRIVER cbid=9 [ 158732164 - 158732220 ] process 2743138, thread 1787117568, correlation 514
DRIVER cbid=9 [ 158732277 - 158732332 ] process 2743138, thread 1787117568, correlation 515
DRIVER cbid=9 [ 158732391 - 158732611 ] process 2743138, thread 1787117568, correlation 516
DRIVER cbid=9 [ 158732669 - 158732725 ] process 2743138, thread 1787117568, correlation 517
DRIVER cbid=9 [ 158732782 - 158732947 ] process 2743138, thread 1787117568, correlation 518
DRIVER cbid=9 [ 158733007 - 158733047 ] process 2743138, thread 1787117568, correlation 519
DRIVER cbid=9 [ 158733105 - 158733167 ] process 2743138, thread 1787117568, correlation 520
DRIVER cbid=9 [ 158733226 - 158733276 ] process 2743138, thread 1787117568, correlation 521
DRIVER cbid=482 [ 158733486 - 158733548 ] process 2743138, thread 1787117568, correlation 522
DRIVER cbid=9 [ 158733611 - 158733657 ] process 2743138, thread 1787117568, correlation 523
DRIVER cbid=9 [ 158733717 - 158733764 ] process 2743138, thread 1787117568, correlation 524
DRIVER cbid=9 [ 158733830 - 158733887 ] process 2743138, thread 1787117568, correlation 525
DRIVER cbid=9 [ 158733945 - 158733994 ] process 2743138, thread 1787117568, correlation 526
DRIVER cbid=9 [ 158734053 - 158734327 ] process 2743138, thread 1787117568, correlation 527
DRIVER cbid=9 [ 158734389 - 158734460 ] process 2743138, thread 1787117568, correlation 528
DRIVER cbid=9 [ 158734519 - 158734586 ] process 2743138, thread 1787117568, correlation 529
DRIVER cbid=9 [ 158734646 - 158734786 ] process 2743138, thread 1787117568, correlation 530
DRIVER cbid=9 [ 158734846 - 158734912 ] process 2743138, thread 1787117568, correlation 531
DRIVER cbid=9 [ 158734972 - 158735033 ] process 2743138, thread 1787117568, correlation 532
DRIVER cbid=9 [ 158735092 - 158735148 ] process 2743138, thread 1787117568, correlation 533
DRIVER cbid=9 [ 158735206 - 158735312 ] process 2743138, thread 1787117568, correlation 534
DRIVER cbid=9 [ 158735373 - 158735429 ] process 2743138, thread 1787117568, correlation 535
DRIVER cbid=9 [ 158735488 - 158735545 ] process 2743138, thread 1787117568, correlation 536
DRIVER cbid=9 [ 158735600 - 158735736 ] process 2743138, thread 1787117568, correlation 537
DRIVER cbid=9 [ 158735797 - 158735852 ] process 2743138, thread 1787117568, correlation 538
RUNTIME cbid=3 [ 158770310 - 158770567 ] process 2743138, thread 1787117568, correlation 539
DRIVER cbid=3 [ 158770778 - 158770844 ] process 2743138, thread 1787117568, correlation 540
DRIVER cbid=5 [ 158770972 - 158772619 ] process 2743138, thread 1787117568, correlation 541
OVERHEAD INSTRUMENTATION [ 248972658, 248996452 ] THREAD id 1787117568
OVERHEAD INSTRUMENTATION [ 249046865, 249061567 ] THREAD id 1787117568
OVERHEAD RESOURCE [ 249061750, 249065842 ] THREAD id 1787117568
CONTEXT 1, device 0, compute API CUDA, NULL stream 7
OVERHEAD RESOURCE [ 249070688, 250393479 ] THREAD id 1787117568
OVERHEAD RESOURCE [ 250393935, 251747474 ] THREAD id 1787117568
OVERHEAD RESOURCE [ 251747759, 253150354 ] THREAD id 1787117568
RUNTIME cbid=16 [ 158779767 - 253174037 ] process 2743138, thread 1787117568, correlation 542
RUNTIME cbid=20 [ 253180459 - 253241928 ] process 2743138, thread 1787117568, correlation 543
RUNTIME cbid=20 [ 253242360 - 253246815 ] process 2743138, thread 1787117568, correlation 544
RUNTIME cbid=20 [ 253246941 - 253248747 ] process 2743138, thread 1787117568, correlation 545
MEMCPY HtoD [ 253285822 - 253314142 ] device 0, context 1, stream 7, correlation 546/r0
RUNTIME cbid=41 [ 253251438 - 253284650 ] process 2743138, thread 1787117568, correlation 546
MEMCPY HtoD [ 253315326 - 253337022 ] device 0, context 1, stream 7, correlation 547/r0
RUNTIME cbid=41 [ 253284772 - 253310997 ] process 2743138, thread 1787117568, correlation 547
OVERHEAD [ 253313183, 257958725 ] THREAD id 1787117568
OVERHEAD [ 257960584, 257960883 ] THREAD id 1787117568
RUNTIME cbid=211 [ 253312023 - 257962257 ] process 2743138, thread 1787117568, correlation 548
RUNTIME cbid=211 [ 257962924 - 257963305 ] process 2743138, thread 1787117568, correlation 549
MEMCPY DtoH [ 257973505 - 257999810 ] device 0, context 1, stream 7, correlation 550/r0
RUNTIME cbid=41 [ 257963460 - 258063812 ] process 2743138, thread 1787117568, correlation 550
RUNTIME cbid=165 [ 258064166 - 258066143 ] process 2743138, thread 1787117568, correlation 551
RUNTIME cbid=22 [ 258066892 - 258071243 ] process 2743138, thread 1787117568, correlation 552
RUNTIME cbid=22 [ 258071331 - 258073397 ] process 2743138, thread 1787117568, correlation 553
RUNTIME cbid=22 [ 258073501 - 258135083 ] process 2743138, thread 1787117568, correlation 554
RUNTIME cbid=129 [ 258135526 - 258147924 ] process 2743138, thread 1787117568, correlation 555
RUNTIME cbid=20 [ 258148395 - 258197445 ] process 2743138, thread 1787117568, correlation 556
RUNTIME cbid=20 [ 258197767 - 258200361 ] process 2743138, thread 1787117568, correlation 557
RUNTIME cbid=20 [ 258200449 - 258202406 ] process 2743138, thread 1787117568, correlation 558
MEMCPY HtoD [ 258223490 - 258252194 ] device 0, context 1, stream 13, correlation 559/r0
RUNTIME cbid=41 [ 258202674 - 258222169 ] process 2743138, thread 1787117568, correlation 559
MEMCPY HtoD [ 258253346 - 258280866 ] device 0, context 1, stream 13, correlation 560/r0
RUNTIME cbid=41 [ 258222340 - 258237171 ] process 2743138, thread 1787117568, correlation 560
RUNTIME cbid=211 [ 258237777 - 258238314 ] process 2743138, thread 1787117568, correlation 561
RUNTIME cbid=211 [ 258238549 - 258238684 ] process 2743138, thread 1787117568, correlation 562
MEMCPY DtoH [ 258285858 - 258311682 ] device 0, context 1, stream 13, correlation 563/r0
RUNTIME cbid=41 [ 258238777 - 258327325 ] process 2743138, thread 1787117568, correlation 563
RUNTIME cbid=131 [ 258327842 - 258329279 ] process 2743138, thread 1787117568, correlation 564
RUNTIME cbid=22 [ 258329778 - 258332990 ] process 2743138, thread 1787117568, correlation 565
RUNTIME cbid=22 [ 258333081 - 258334716 ] process 2743138, thread 1787117568, correlation 566
RUNTIME cbid=22 [ 258334838 - 258372048 ] process 2743138, thread 1787117568, correlation 567
RUNTIME cbid=165 [ 258372421 - 258373941 ] process 2743138, thread 1787117568, correlation 568
OVERHEAD RESOURCE [ 258377284, 258380474 ] THREAD id 1787117568
OVERHEAD BUFFER_FLUSH [ 258380787, 258395488 ] THREAD id 1787117568
RUNTIME cbid=164 [ 258374083 - 310954063 ] process 2743138, thread 1787117568, correlation 569

its really weird that it output this message DRIVER cbid=677 [ 155580247 - 155583215 ] process 2743138, thread 1787117568, correlation 1continuously，and i cant find any thing related to the two kernels executed in the main().

so can you help me check this problem?very very thanks!

mjain · June 24, 2025, 1:15pm

The output shows CUDA API trace for kernel launches (cbid 211), but it is unclear why the kernel record is not generated.

RUNTIME cbid=211 [ 253312023 - 257962257 ] process 2743138, thread 1787117568, correlation 548
RUNTIME cbid=211 [ 257962924 - 257963305 ] process 2743138, thread 1787117568, correlation 549

Please try using the sample activity_trace_async, as the activity_trace sample was dropped a long time ago.

fwd20001101 · June 25, 2025, 5:44am

Thanks for your reply,and follow your suggestion,i tried to compile and run`activity_trace_async in the sample code,and this is what i got:

DRIVER [ 1750829693779065574, 1750829693779065652 ] duration 78, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 118
DRIVER [ 1750829693779065723, 1750829693779065793 ] duration 70, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 119
DRIVER [ 1750829693779065864, 1750829693779065933 ] duration 69, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 120
DRIVER [ 1750829693779066003, 1750829693779066065 ] duration 62, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 121
DRIVER [ 1750829693779066126, 1750829693779066196 ] duration 70, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 122
DRIVER [ 1750829693779066265, 1750829693779066335 ] duration 70, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 123
DRIVER [ 1750829693779066461, 1750829693779066546 ] duration 85, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 124
DRIVER [ 1750829693779066618, 1750829693779066748 ] duration 130, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 125
DRIVER [ 1750829693779066818, 1750829693779066881 ] duration 63, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 126
DRIVER [ 1750829693779066946, 1750829693779067007 ] duration 61, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 127
DRIVER [ 1750829693779067074, 1750829693779067133 ] duration 59, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 128
DRIVER [ 1750829693779132814, 1750829693779136225 ] duration 3411, “cuDeviceGetName”, cbid 5, processId 3800730, threadId 318861312, correlationId 425
DRIVER [ 1750829693779136504, 1750829693779139408 ] duration 2904, “cuDeviceTotalMem_v2”, cbid 259, processId 3800730, threadId 318861312, correlationId 426
DRIVER [ 1750829693779139609, 1750829693779139835 ] duration 226, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 427
DRIVER [ 1750829693779139919, 1750829693779139980 ] duration 61, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 428
DRIVER [ 1750829693779140053, 1750829693779140115 ] duration 62, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 429
DRIVER [ 1750829693779140186, 1750829693779140256 ] duration 70, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 430
DRIVER [ 1750829693779140325, 1750829693779140423 ] duration 98, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 431
DRIVER [ 1750829693779140499, 1750829693779145656 ] duration 5157, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 432
DRIVER [ 1750829693779145788, 1750829693779145843 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 433
DRIVER [ 1750829693779145919, 1750829693779145969 ] duration 50, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 434
DRIVER [ 1750829693779146044, 1750829693779146095 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 435
DRIVER [ 1750829693779146168, 1750829693779146220 ] duration 52, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 436
DRIVER [ 1750829693779146299, 1750829693779146462 ] duration 163, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 437
DRIVER [ 1750829693779146532, 1750829693779146579 ] duration 47, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 438
DRIVER [ 1750829693779146657, 1750829693779146716 ] duration 59, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 439
DRIVER [ 1750829693779146787, 1750829693779146836 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 440
DRIVER [ 1750829693779146905, 1750829693779146952 ] duration 47, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 441
DRIVER [ 1750829693779147023, 1750829693779147069 ] duration 46, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 442
DRIVER [ 1750829693779147137, 1750829693779147186 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 443
DRIVER [ 1750829693779147253, 1750829693779147301 ] duration 48, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 444
DRIVER [ 1750829693779147371, 1750829693779147429 ] duration 58, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 445
DRIVER [ 1750829693779147496, 1750829693779147543 ] duration 47, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 446
DRIVER [ 1750829693779147611, 1750829693779147658 ] duration 47, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 447
DRIVER [ 1750829693779147725, 1750829693779147780 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 448
DRIVER [ 1750829693779147846, 1750829693779147903 ] duration 57, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 449
DRIVER [ 1750829693779147969, 1750829693779148026 ] duration 57, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 450
DRIVER [ 1750829693779148093, 1750829693779148147 ] duration 54, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 451
DRIVER [ 1750829693779148213, 1750829693779148270 ] duration 57, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 452
DRIVER [ 1750829693779148338, 1750829693779148393 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 453
DRIVER [ 1750829693779148461, 1750829693779148516 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 454
DRIVER [ 1750829693779148583, 1750829693779148638 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 455
DRIVER [ 1750829693779148705, 1750829693779148761 ] duration 56, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 456
DRIVER [ 1750829693779148827, 1750829693779148882 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 457
DRIVER [ 1750829693779148950, 1750829693779149003 ] duration 53, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 458
DRIVER [ 1750829693779149071, 1750829693779149136 ] duration 65, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 459
DRIVER [ 1750829693779149204, 1750829693779149258 ] duration 54, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 460
DRIVER [ 1750829693779149388, 1750829693779149442 ] duration 54, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 461
DRIVER [ 1750829693779149509, 1750829693779149563 ] duration 54, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 462
DRIVER [ 1750829693779149629, 1750829693779149685 ] duration 56, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 463
DRIVER [ 1750829693779149753, 1750829693779149808 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 464
DRIVER [ 1750829693779149876, 1750829693779149931 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 465
DRIVER [ 1750829693779149998, 1750829693779150053 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 466
DRIVER [ 1750829693779150120, 1750829693779150175 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 467
DRIVER [ 1750829693779150242, 1750829693779150297 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 468
DRIVER [ 1750829693779150363, 1750829693779150418 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 469
DRIVER [ 1750829693779150485, 1750829693779150541 ] duration 56, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 470
DRIVER [ 1750829693779150607, 1750829693779150663 ] duration 56, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 471
DRIVER [ 1750829693779150729, 1750829693779150785 ] duration 56, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 472
DRIVER [ 1750829693779150850, 1750829693779150905 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 473
DRIVER [ 1750829693779150971, 1750829693779151024 ] duration 53, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 474
DRIVER [ 1750829693779151091, 1750829693779151140 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 475
DRIVER [ 1750829693779151207, 1750829693779151256 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 476
DRIVER [ 1750829693779151322, 1750829693779151376 ] duration 54, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 477
DRIVER [ 1750829693779151442, 1750829693779151498 ] duration 56, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 478
DRIVER [ 1750829693779151567, 1750829693779151621 ] duration 54, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 479
DRIVER [ 1750829693779151689, 1750829693779151744 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 480
DRIVER [ 1750829693779151811, 1750829693779151866 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 481
DRIVER [ 1750829693779151933, 1750829693780597607 ] duration 1445674, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 482
DRIVER [ 1750829693780597786, 1750829693780597858 ] duration 72, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 483
DRIVER [ 1750829693780597932, 1750829693780597983 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 484
DRIVER [ 1750829693780598054, 1750829693780598106 ] duration 52, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 485
DRIVER [ 1750829693780598175, 1750829693780598226 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 486
DRIVER [ 1750829693780598304, 1750829693780598356 ] duration 52, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 487
DRIVER [ 1750829693780598418, 1750829693780598473 ] duration 55, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 488
DRIVER [ 1750829693780598536, 1750829693780598587 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 489
DRIVER [ 1750829693780598642, 1750829693780598693 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 490
DRIVER [ 1750829693780598754, 1750829693780598805 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 491
DRIVER [ 1750829693780598867, 1750829693780598915 ] duration 48, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 492
DRIVER [ 1750829693780598975, 1750829693780599024 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 493
DRIVER [ 1750829693780599086, 1750829693780599135 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 494
DRIVER [ 1750829693780599196, 1750829693780599245 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 495
DRIVER [ 1750829693780599306, 1750829693780599413 ] duration 107, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 496
DRIVER [ 1750829693780599474, 1750829693780599522 ] duration 48, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 497
DRIVER [ 1750829693780599583, 1750829693780599631 ] duration 48, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 498
DRIVER [ 1750829693780599692, 1750829693780599740 ] duration 48, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 499
DRIVER [ 1750829693780599800, 1750829693780599870 ] duration 70, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 500
DRIVER [ 1750829693780599930, 1750829693780599978 ] duration 48, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 501
DRIVER [ 1750829693780600031, 1750829693780600080 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 502
DRIVER [ 1750829693780600140, 1750829693780600188 ] duration 48, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 503
DRIVER [ 1750829693780600251, 1750829693781926741 ] duration 1326490, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 504
DRIVER [ 1750829693781926884, 1750829693781926966 ] duration 82, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 505
DRIVER [ 1750829693781927036, 1750829693781927144 ] duration 108, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 506
DRIVER [ 1750829693781927212, 1750829693781927263 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 507
DRIVER [ 1750829693781927322, 1750829693781927371 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 508
DRIVER [ 1750829693781927428, 1750829693781927489 ] duration 61, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 509
DRIVER [ 1750829693781927551, 1750829693781927601 ] duration 50, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 510
DRIVER [ 1750829693781927671, 1750829693781927721 ] duration 50, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 511
DRIVER [ 1750829693781927782, 1750829693781928303 ] duration 521, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 512
DRIVER [ 1750829693781928385, 1750829693781928446 ] duration 61, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 513
DRIVER [ 1750829693781928507, 1750829693781928556 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 514
DRIVER [ 1750829693781928627, 1750829693781928676 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 515
DRIVER [ 1750829693781928746, 1750829693781928937 ] duration 191, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 516
DRIVER [ 1750829693781928997, 1750829693781929046 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 517
DRIVER [ 1750829693781929107, 1750829693781929268 ] duration 161, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 518
DRIVER [ 1750829693781929337, 1750829693781929383 ] duration 46, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 519
DRIVER [ 1750829693781929442, 1750829693781929493 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 520
DRIVER [ 1750829693781929553, 1750829693781929594 ] duration 41, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 521
DRIVER [ 1750829693781929858, 1750829693781929929 ] duration 71, “cuDeviceGetUuid”, cbid 482, processId 3800730, threadId 318861312, correlationId 522
DRIVER [ 1750829693781930003, 1750829693781930056 ] duration 53, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 523
DRIVER [ 1750829693781930126, 1750829693781930176 ] duration 50, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 524
DRIVER [ 1750829693781930242, 1750829693781930289 ] duration 47, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 525
DRIVER [ 1750829693781930353, 1750829693781930402 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 526
DRIVER [ 1750829693781930461, 1750829693781930866 ] duration 405, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 527
DRIVER [ 1750829693781930925, 1750829693781930979 ] duration 54, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 528
DRIVER [ 1750829693781931038, 1750829693781931088 ] duration 50, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 529
DRIVER [ 1750829693781931149, 1750829693781931280 ] duration 131, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 530
DRIVER [ 1750829693781931339, 1750829693781931386 ] duration 47, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 531
DRIVER [ 1750829693781931445, 1750829693781931497 ] duration 52, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 532
DRIVER [ 1750829693781931559, 1750829693781931606 ] duration 47, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 533
DRIVER [ 1750829693781931660, 1750829693781931766 ] duration 106, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 534
DRIVER [ 1750829693781931829, 1750829693781931878 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 535
DRIVER [ 1750829693781931939, 1750829693781931990 ] duration 51, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 536
DRIVER [ 1750829693781932044, 1750829693781932180 ] duration 136, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 537
DRIVER [ 1750829693781932241, 1750829693781932290 ] duration 49, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 538
RUNTIME [ 1750829693781966587, 1750829693781966857 ] duration 270, “cudaGetDeviceCount_v3020”, cbid 3, processId 3800730, threadId 318861312, correlationId 539
DRIVER [ 1750829693781967149, 1750829693781967199 ] duration 50, “cuDeviceGet”, cbid 3, processId 3800730, threadId 318861312, correlationId 540
DRIVER [ 1750829693781967321, 1750829693781969156 ] duration 1835, “cuDeviceGetName”, cbid 5, processId 3800730, threadId 318861312, correlationId 541
OVERHEAD CUPTI_INSTRUMENTATION [ 1750829693880744001, 1750829693880768699 ] duration 24698, THREAD, id 318861312, correlation id 542
OVERHEAD CUPTI_INSTRUMENTATION [ 1750829693880822830, 1750829693880839657 ] duration 16827, THREAD, id 318861312, correlation id 542
OVERHEAD CUPTI_RESOURCE [ 1750829693880839759, 1750829693880843038 ] duration 3279, THREAD, id 318861312, correlation id 542
CONTEXT computeApiKind CUDA, contextId 1, deviceId 0, nullStreamId 7, CIG mode 0
OVERHEAD CUPTI_RESOURCE [ 1750829693880847947, 1750829693882174584 ] duration 1326637, THREAD, id 318861312, correlation id 542
OVERHEAD CUPTI_RESOURCE [ 1750829693882174991, 1750829693883535604 ] duration 1360613, THREAD, id 318861312, correlation id 542
OVERHEAD CUPTI_RESOURCE [ 1750829693883535945, 1750829693885078471 ] duration 1542526, THREAD, id 318861312, correlation id 542
RUNTIME [ 1750829693781977381, 1750829693885102083 ] duration 103124702, “cudaSetDevice_v3020”, cbid 16, processId 3800730, threadId 318861312, correlationId 542
RUNTIME [ 1750829693885106120, 1750829693885170200 ] duration 64080, “cudaMalloc_v3020”, cbid 20, processId 3800730, threadId 318861312, correlationId 543
RUNTIME [ 1750829693885170657, 1750829693885173426 ] duration 2769, “cudaMalloc_v3020”, cbid 20, processId 3800730, threadId 318861312, correlationId 544
RUNTIME [ 1750829693885173547, 1750829693885175404 ] duration 1857, “cudaMalloc_v3020”, cbid 20, processId 3800730, threadId 318861312, correlationId 545
MEMCPY “HtoD” [ 1750829693885213918, 1750829693885242078 ] duration 28160, size 200000, copyCount 1, srcKind PAGEABLE, dstKind DEVICE, correlationId 546
deviceId 0, contextId 1, streamId 7, graphId 0, graphNodeId 0, channelId 8, channelType ASYNC_MEMCPY
RUNTIME [ 1750829693885178493, 1750829693885212963 ] duration 34470, “cudaMemcpyAsync_v3020”, cbid 41, processId 3800730, threadId 318861312, correlationId 546
MEMCPY “HtoD” [ 1750829693885247166, 1750829693885281566 ] duration 34400, size 200000, copyCount 1, srcKind PAGEABLE, dstKind DEVICE, correlationId 547
deviceId 0, contextId 1, streamId 7, graphId 0, graphNodeId 0, channelId 8, channelType ASYNC_MEMCPY
RUNTIME [ 1750829693885213139, 1750829693885245957 ] duration 32818, “cudaMemcpyAsync_v3020”, cbid 41, processId 3800730, threadId 318861312, correlationId 547
OVERHEAD RUNTIME_TRIGGERED_MODULE_LOADING [ 1750829693885248908, 1750829693885307342 ] duration 58434, THREAD, id 318861312, correlation id 548
OVERHEAD RUNTIME_TRIGGERED_MODULE_LOADING [ 1750829693885308389, 1750829693885324828 ] duration 16439, THREAD, id 318861312, correlation id 548
OVERHEAD LAZY_FUNCTION_LOADING [ 1750829693885325973, 1750829693885334289 ] duration 8316, THREAD, id 318861312, correlation id 548
CONCURRENT_KERNEL [ 1750829693885351710, 1750829693885354270 ] duration 2560, “_Z9VectorAddPKiS0_Pii”, correlationId 548, cacheConfigRequested 0, cacheConfigExecuted 0
grid [ 196, 1, 1 ], block [ 256, 1, 1 ], cluster [ 0, 0, 0 ], sharedMemory (static 0, dynamic 0)
deviceId 0, contextId 1, streamId 7, graphId 0, graphNodeId 0, channelId 0, channelType COMPUTE
RUNTIME [ 1750829693885247297, 1750829693885351381 ] duration 104084, “cudaLaunchKernel_v7000”, cbid 211, processId 3800730, threadId 318861312, correlationId 548
RUNTIME [ 1750829693885351763, 1750829693885352033 ] duration 270, “cudaGetLastError_v3020”, cbid 10, processId 3800730, threadId 318861312, correlationId 549
OVERHEAD LAZY_FUNCTION_LOADING [ 1750829693885353068, 1750829693885359214 ] duration 6146, THREAD, id 318861312, correlation id 550
CONCURRENT_KERNEL [ 1750829693885364798, 1750829693885366782 ] duration 1984, “_Z14VectorSubtractPKiS0_Pii”, correlationId 550, cacheConfigRequested 0, cacheConfigExecuted 0
grid [ 196, 1, 1 ], block [ 256, 1, 1 ], cluster [ 0, 0, 0 ], sharedMemory (static 0, dynamic 0)
deviceId 0, contextId 1, streamId 7, graphId 0, graphNodeId 0, channelId 0, channelType COMPUTE
RUNTIME [ 1750829693885352575, 1750829693885362802 ] duration 10227, “cudaLaunchKernel_v7000”, cbid 211, processId 3800730, threadId 318861312, correlationId 550
RUNTIME [ 1750829693885363036, 1750829693885363109 ] duration 73, “cudaGetLastError_v3020”, cbid 10, processId 3800730, threadId 318861312, correlationId 551
MEMCPY “DtoH” [ 1750829693885371550, 1750829693885397597 ] duration 26047, size 200000, copyCount 1, srcKind DEVICE, dstKind PAGEABLE, correlationId 552
deviceId 0, contextId 1, streamId 7, graphId 0, graphNodeId 0, channelId 12, channelType ASYNC_MEMCPY
RUNTIME [ 1750829693885363255, 1750829693885466494 ] duration 103239, “cudaMemcpyAsync_v3020”, cbid 41, processId 3800730, threadId 318861312, correlationId 552
RUNTIME [ 1750829693885466866, 1750829693885469195 ] duration 2329, “cudaDeviceSynchronize_v3020”, cbid 165, processId 3800730, threadId 318861312, correlationId 553
RUNTIME [ 1750829693885469829, 1750829693885474065 ] duration 4236, “cudaFree_v3020”, cbid 22, processId 3800730, threadId 318861312, correlationId 554
RUNTIME [ 1750829693885474160, 1750829693885476046 ] duration 1886, “cudaFree_v3020”, cbid 22, processId 3800730, threadId 318861312, correlationId 555
RUNTIME [ 1750829693885476168, 1750829693885531235 ] duration 55067, “cudaFree_v3020”, cbid 22, processId 3800730, threadId 318861312, correlationId 556
RUNTIME [ 1750829693885531745, 1750829693885560949 ] duration 29204, “cudaStreamCreate_v3020”, cbid 129, processId 3800730, threadId 318861312, correlationId 557
RUNTIME [ 1750829693885563314, 1750829693885607127 ] duration 43813, “cudaMalloc_v3020”, cbid 20, processId 3800730, threadId 318861312, correlationId 558
RUNTIME [ 1750829693885607468, 1750829693885609813 ] duration 2345, “cudaMalloc_v3020”, cbid 20, processId 3800730, threadId 318861312, correlationId 559
RUNTIME [ 1750829693885609928, 1750829693885611534 ] duration 1606, “cudaMalloc_v3020”, cbid 20, processId 3800730, threadId 318861312, correlationId 560
MEMCPY “HtoD” [ 1750829693885624573, 1750829693885653693 ] duration 29120, size 200000, copyCount 1, srcKind PAGEABLE, dstKind DEVICE, correlationId 561
deviceId 0, contextId 1, streamId 13, graphId 0, graphNodeId 0, channelId 10, channelType ASYNC_MEMCPY
RUNTIME [ 1750829693885611811, 1750829693885622949 ] duration 11138, “cudaMemcpyAsync_v3020”, cbid 41, processId 3800730, threadId 318861312, correlationId 561
MEMCPY “HtoD” [ 1750829693885655581, 1750829693885682909 ] duration 27328, size 200000, copyCount 1, srcKind PAGEABLE, dstKind DEVICE, correlationId 562
deviceId 0, contextId 1, streamId 13, graphId 0, graphNodeId 0, channelId 10, channelType ASYNC_MEMCPY
RUNTIME [ 1750829693885623127, 1750829693885633339 ] duration 10212, “cudaMemcpyAsync_v3020”, cbid 41, processId 3800730, threadId 318861312, correlationId 562
CONCURRENT_KERNEL [ 1750829693885692157, 1750829693885694269 ] duration 2112, “_Z9VectorAddPKiS0_Pii”, correlationId 563, cacheConfigRequested 0, cacheConfigExecuted 0
grid [ 196, 1, 1 ], block [ 256, 1, 1 ], cluster [ 0, 0, 0 ], sharedMemory (static 0, dynamic 0)
deviceId 0, contextId 1, streamId 13, graphId 0, graphNodeId 0, channelId 1, channelType COMPUTE
RUNTIME [ 1750829693885633925, 1750829693885641065 ] duration 7140, “cudaLaunchKernel_v7000”, cbid 211, processId 3800730, threadId 318861312, correlationId 563
RUNTIME [ 1750829693885641279, 1750829693885641398 ] duration 119, “cudaGetLastError_v3020”, cbid 10, processId 3800730, threadId 318861312, correlationId 564
CONCURRENT_KERNEL [ 1750829693885695069, 1750829693885697053 ] duration 1984, “_Z14VectorSubtractPKiS0_Pii”, correlationId 565, cacheConfigRequested 0, cacheConfigExecuted 0
grid [ 196, 1, 1 ], block [ 256, 1, 1 ], cluster [ 0, 0, 0 ], sharedMemory (static 0, dynamic 0)
deviceId 0, contextId 1, streamId 13, graphId 0, graphNodeId 0, channelId 1, channelType COMPUTE
RUNTIME [ 1750829693885641702, 1750829693885644207 ] duration 2505, “cudaLaunchKernel_v7000”, cbid 211, processId 3800730, threadId 318861312, correlationId 565
RUNTIME [ 1750829693885644350, 1750829693885644393 ] duration 43, “cudaGetLastError_v3020”, cbid 10, processId 3800730, threadId 318861312, correlationId 566
MEMCPY “DtoH” [ 1750829693885704317, 1750829693885732509 ] duration 28192, size 200000, copyCount 1, srcKind DEVICE, dstKind PAGEABLE, correlationId 567
deviceId 0, contextId 1, streamId 13, graphId 0, graphNodeId 0, channelId 13, channelType ASYNC_MEMCPY
RUNTIME [ 1750829693885644544, 1750829693885803046 ] duration 158502, “cudaMemcpyAsync_v3020”, cbid 41, processId 3800730, threadId 318861312, correlationId 567
RUNTIME [ 1750829693885803395, 1750829693885804862 ] duration 1467, “cudaStreamSynchronize_v3020”, cbid 131, processId 3800730, threadId 318861312, correlationId 568
RUNTIME [ 1750829693885805338, 1750829693885808780 ] duration 3442, “cudaFree_v3020”, cbid 22, processId 3800730, threadId 318861312, correlationId 569
RUNTIME [ 1750829693885808869, 1750829693885810536 ] duration 1667, “cudaFree_v3020”, cbid 22, processId 3800730, threadId 318861312, correlationId 570
RUNTIME [ 1750829693885810631, 1750829693885849994 ] duration 39363, “cudaFree_v3020”, cbid 22, processId 3800730, threadId 318861312, correlationId 571
RUNTIME [ 1750829693885850395, 1750829693885851911 ] duration 1516, “cudaDeviceSynchronize_v3020”, cbid 165, processId 3800730, threadId 318861312, correlationId 572
OVERHEAD CUPTI_BUFFER_FLUSH [ 1750829693885911909, 1750829693885928611 ] duration 16702, THREAD, id 249561088, correlation id 0
OVERHEAD CUPTI_BUFFER_FLUSH [ 1750829693885982812, 1750829693885987897 ] duration 5085, THREAD, id 249561088, correlation id 0
Calling CUPTI API: cuptiUnsubscribe(globals.subscriberHandle)
Calling CUPTI API: cuptiActivityFlushAll(1)
OVERHEAD CUPTI_RESOURCE [ 1750829693909798634, 1750829693909809123 ] duration 10489, THREAD, id 318861312, correlation id 573
OVERHEAD ACTIVITY_BUFFER_REQUEST [ 1750829693909810834, 1750829693912032046 ] duration 2221212, THREAD, id 318861312, correlation id 573
RUNTIME [ 1750829693909781756, 1750829693969130488 ] duration 59348732, “cudaDeviceReset_v3020”, cbid 164, processId 3800730, threadId 318861312, correlationId 573

Did the kernel ran?
And im confused that why these two message keeps appearing,in normal logic,these two APIs should be called only once?

DRIVER [ 1750829693779066946, 1750829693779067007 ] duration 61, “cuGetProcAddress_v2”, cbid 677, processId 3800730, threadId 318861312, correlationId 127

DRIVER [ 1750829693781926884, 1750829693781926966 ] duration 82, “cuDeviceGetAttribute”, cbid 9, processId 3800730, threadId 318861312, correlationId 505

thanks for your help!

Sanjiv.Satoor · June 25, 2025, 8:18am

CONCURRENT_KERNEL activity in the output shows that kernel ran

Sanjiv.Satoor · June 25, 2025, 8:20am

“cuGetProcAddress_v2” and “cuDeviceGetAttribute” CUDA Driver APIs are called multiple times for most CUDA applications - so this is expected.

fwd20001101 · June 25, 2025, 9:05am

So,did it mean cuGetProcAddress_v2 and cuDeviceGetAttributeare called several times,and has nothing to do with what i wrote in my code,which means even if i only call those APIs in my code once,these APIs still run multiple times?

Sanjiv.Satoor · June 26, 2025, 10:18am

The cuGetProcAddress_v2 and cuDeviceGetAttribute APIs can be called by other library calls. We currently do not have to only trace CUDA APIs called from user code.
CUPTI provides a way to selectively enable and disable tracing of CUDA APIs from 12.8+ using cuptiActivityEnableRuntimeApi and cuptiActivityEnableDriverApi APIs. These could be used to skip tracing specific APIs e.g. cuGetProcAddress_v2 and cuDeviceGetAttribute.

veraj · July 28, 2025, 8:42am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
CUPTI Sample tutorial wrong CUPTI – CUDA Profiler Tools Interface cuda , kernel	2	170	July 28, 2025
How can i linking the CUDA driver API library and CUPTI library on google colab? CUPTI – CUDA Profiler Tools Interface cuda	1	2040	June 1, 2021
How to use these API cuptiKernelReplaySubscribeUpdate cuptiEnableKernelReplayMode cuptiDisableKernelReplayMode CUPTI – CUDA Profiler Tools Interface	3	673	June 21, 2023
Measure metrics from a CUDA binary? CUPTI – CUDA Profiler Tools Interface cuda , kernel	2	791	October 12, 2021
CUPTI Samples FAIL CUPTI – CUDA Profiler Tools Interface cuda	4	713	August 31, 2020
CUDA Profiler Tools Interface (CUPTI) for CUDA Toolkit 12.5 is now available CUPTI – CUDA Profiler Tools Interface	6	493	October 23, 2024
CUDA Profiler Tools Interface (CUPTI) for CUDA Toolkit 12.4 is now available CUPTI – CUDA Profiler Tools Interface	7	431	October 23, 2024
Profiling cuda graph with CUPTI Profiling API CUPTI – CUDA Profiler Tools Interface tensorrt	4	1408	September 29, 2025
CUDA Profiler Tools Interface (CUPTI) for CUDA Toolkit 12.0 is now available CUPTI – CUDA Profiler Tools Interface	5	1657	October 23, 2024
NVIDIA® CUDA Profiler Tools Interface (CUPTI) 2019.1 is now available CUPTI – CUDA Profiler Tools Interface	0	1940	March 1, 2019

CUPTI Sample Code problem

Related topics