cuda stream high priority could not always schedule high prority

Accelerated Computing CUDA CUDA Programming and Performance
1

In c++ project, we have a high priority stream. most of time it can be schedule first, but sometimes it’s not.

runtime prof as below

[url]http://chuantu.xyz/t6/702/1562576003x992245975.png[/url]
if you could not visit this url， the attachment is the same picture。
Is there any idea？ Thanks for your answer。

cuda-problem.png
cuda-problem.png2210×922 424 KB

2

It might be that you have a misplaced cudaStreamSynchronize() that is preventing your higher priority stream kernel from getting scheduled, as your picture seems to be suggesting. There is really no way to confirm this from a static picture of the profiler output.

It doesn’t seem to apply to your case, but you should also be aware that depending on the GPU, blocks from high priority streams must wait for available space on the GPU SMs. If blocks from low priority streams get scheduled first (e.g. because they were launched first), and they fully occupy an SM, then subsequently issued blocks from higher priority streams may have to wait until space (resources) is available on the SM, before they can be scheduled.

3

Hi, we use TensorRT as inference framework. And we calling enqueue() from multiple host threads.
When we create shuffle layer as below,
nvinfer1::IShuffleLayer *shuffleLayer = net->addShuffle(*inputs[0]);
shuffleLayer->setFirstTranspose(perm);
shuffleLayer->setName(layer_param.name().c_str());
shuffleLayer->setReshapeDimensions(dims);

We found enqueue() kernels in multi-thread which use multi-stream can not execute parallel in GPU.
The question is TensorRT shuffle layer preventing other stream kernel scheduled？