I’m working on a project where I have to process multiple images (either 2 or 4) as fast as possible in a CNN.
At the moment I’m just batching the images and and doing batch-inference in a single IExecutionContext.
I was wondering if it would be possible to create independent Execution Contexts and run them concurrently.
So my Questions are:
- Instead of batching, can I create multiple Execution Contexts from the same engine?
2.) Will they run concurrently on a single GPU if I use them from seperate threads?
3.) Will this be faster than batching the inputs?
Thank you for your answers.