Multiple concurrent Execution Contexts?

I’m working on a project where I have to process multiple images (either 2 or 4) as fast as possible in a CNN.
At the moment I’m just batching the images and and doing batch-inference in a single IExecutionContext.
I was wondering if it would be possible to create independent Execution Contexts and run them concurrently.
So my Questions are:

  1. Instead of batching, can I create multiple Execution Contexts from the same engine?
    2.) Will they run concurrently on a single GPU if I use them from seperate threads?
    3.) Will this be faster than batching the inputs?

Thank you for your answers.

Batching will give you higher throughput.
Increase batch to fully use GPU resource for better performance.


Thank you for your reply.
Just out of curiosity: From your answer I gather that two Execution Contexts wouldn’t actually run in parallel on the GPU?

I will run in parallel and yes, it can have some benefit.
But increase batch to fully use GPU resource might have better performance.


Thank you.
I was just a little confused while batching the input images.
I am experiencing almost no benefit regarding inference time while batching.
For example; When I use batchsize 1 for just a single image my network (A pretty big CNN) will take about 5ms,
In contrast to that, when I use batchsize 4 it will take about 17ms. I was under the impression that batching 4 images wouldn’t take that much longer than inference on a single image.

1 Like

Hello, I want to know how can it run concurrently