nvinfer1::IExecutionContext::enqueue not asynchronous even with cudaStreamNonBlocking stream

[SunilJB]

Hi,

Please refer to below link and examples:

Synchronized inference or Asynchronized inference

Hello, Asynchronous inference execution generally increases performance by overlapping compute as it maximizes GPU utilization. The pros of using execute/sync api is simplicity/code executes inline. The challenge of enqueue/async api is the extra work of handling signal events when the buffers are in use/freed. But you should see more efficient GPU usage with async model. In a typical use case, TensorRT will execute asynchronously. The enqueue() method will add kernels to a CUDA stream spec…

Developer Guide :: NVIDIA Deep Learning TensorRT Documentation

https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/sampleMNIST/sampleMNIST.cpp#L305

Thanks