So I have designed a Profiler using CUPTI Range Profiling API,and I wrote two demos to test Profiler’s performance under Multi-CUDA Streams’ scnerio.
The first demo is to run the same kernel on 4 CUDA Streams created by users
The second demo is to run the same kernel on 4 CUDA Streams,while one of these streams is default stream.
So are there any inferences between streams in these 2 demos?
Actually i got these output messages:
demo1
fangweidong@syh-computed:~/fwd/tensorrt_demo$ env CUDA_INJECTION64_PATH=./libMultiProfiler.so ./test_kernel
Found1devices
Configuring current device0Context0x579d1cd3bea0,Device0+NVIDIA RTX A4000Session 1
Range Name Metric Name Metric Value
0 sm__cycles_elapsed.avg 3029.67
1 sm__cycles_elapsed.avg 3014.67
2 sm__cycles_elapsed.avg 3177.83
3 sm__cycles_elapsed.avg 3025.67
0 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
1 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
2 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
3 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
0 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
1 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
2 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
3 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
Runtime API Name: cudaMemcpyStart Time: 1752548533250313104End Time: 1752548533250322977
Runtime API Duration: 0.009873ms
Runtime API Name: cudaMemcpyStart Time: 1752548533250330907End Time: 1752548533250333100
Runtime API Duration: 0.002193ms
Runtime API Name: cudaMemcpyStart Time: 1752548533250342544End Time: 1752548533250344349
Runtime API Duration: 0.001805ms
Runtime API Name: cudaMemcpyStart Time: 1752548533250347797End Time: 1752548533250349285
Runtime API Duration: 0.001488ms
Runtime API Name: cudaMemcpyStart Time: 1752548533250356704End Time: 1752548533250358447
Runtime API Duration: 0.001743ms
Runtime API Name: cudaMemcpyStart Time: 1752548533250361720End Time: 1752548533250363268
Runtime API Duration: 0.001548ms
Runtime API Name: cudaMemcpyStart Time: 1752548533250369877End Time: 1752548533250371483
Runtime API Duration: 0.001606ms
Runtime API Name: cudaMemcpyStart Time: 1752548533250374657End Time: 1752548533250376113
Runtime API Duration: 0.001456ms
demo2
fangweidong@syh-computed:~/fwd/tensorrt_demo$ env CUDA_INJECTION64_PATH=./libMultiProfiler.so ./default
Found 1 devices
Configuring current device 0Context0x5964f05052d0,Device0+NVIDIA RTX A4000Session 1
Range Name Metric Name Metric Value
0 sm__cycles_elapsed.avg 3028
1 sm__cycles_elapsed.avg 3078.33
2 sm__cycles_elapsed.avg 2988.83
3 sm__cycles_elapsed.avg 2992.83
0 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
1 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
2 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
3 smsp__sass_thread_inst_executed_op_dadd_pred_on.avg 0
0 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
1 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
2 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
3 smsp__sass_thread_inst_executed_op_dfma_pred_on.avg 0
Runtime API Name: cudaMemcpyStart Time: 1752491121161868061End Time: 1752491121161878666
Runtime API Duration: 0.010605ms
Runtime API Name: cudaMemcpyStart Time: 1752491121161884557End Time: 1752491121161887007
Runtime API Duration: 0.00245ms
Runtime API Name: cudaMemcpyStart Time: 1752491121161902017End Time: 1752491121161904566
Runtime API Duration: 0.002549ms
Runtime API Name: cudaMemcpyStart Time: 1752491121161907099End Time: 1752491121161908863
Runtime API Duration: 0.001764ms
Runtime API Name: cudaMemcpyStart Time: 1752491121161913599End Time: 1752491121161915481
Runtime API Duration: 0.001882ms
Runtime API Name: cudaMemcpyStart Time: 1752491121161917725End Time: 1752491121161920063
Runtime API Duration: 0.002338ms
Runtime API Name: cudaMemcpyStart Time: 1752491121161924685End Time: 1752491121161926463
Runtime API Duration: 0.001778ms
Runtime API Name: cudaMemcpyStart Time: 1752491121161928576End Time: 1752491121161930641
Runtime API Duration: 0.002065ms