# Multiple concurrent Execution Contexts?

**URL:** <https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247>\
**Category:** TensorRT\
**Tags:** tensorrt\
**Created:** [May 15, 2020, 9:22am UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247 "2020-05-15T09:22:51Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![CAT](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@CAT](https://forums.developer.nvidia.com/u/CAT)\
**Post date:** [May 15, 2020, 9:22am UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247/1 "2020-05-15T09:22:51Z")

</div>

I’m working on a project where I have to process multiple images (either 2 or 4) as fast as possible in a CNN.  
At the moment I’m just batching the images and and doing batch-inference in a single IExecutionContext.  
I was wondering if it would be possible to create independent Execution Contexts and run them concurrently.  
So my Questions are:

1. Instead of batching, can I create multiple Execution Contexts from the same engine?  
2.) Will they run concurrently on a single GPU if I use them from seperate threads?  
3.) Will this be faster than batching the inputs?

Thank you for your answers.

---

<div class="post-metadata">

**Author:** ![SunilJB](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/suniljb/32/14043_2.png) [@SunilJB](https://forums.developer.nvidia.com/u/SunilJB)\
**Post date:** [May 15, 2020, 5:56pm UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247/2 "2020-05-15T17:56:09Z")

</div>

Hi,  
Batching will give you higher throughput.  
Increase batch to fully use GPU resource for better performance.

Thanks

---

<div class="post-metadata">

**Author:** ![CAT](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@CAT](https://forums.developer.nvidia.com/u/CAT)\
**Post date:** [May 18, 2020, 6:14am UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247/3 "2020-05-18T06:14:38Z")

</div>

Thank you for your reply.  
Just out of curiosity: From your answer I gather that two Execution Contexts wouldn’t actually run in parallel on the GPU?

---

<div class="post-metadata">

**Author:** ![SunilJB](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/suniljb/32/14043_2.png) [@SunilJB](https://forums.developer.nvidia.com/u/SunilJB)\
**Post date:** [May 18, 2020, 6:27am UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247/4 "2020-05-18T06:27:08Z")

</div>

I will run in parallel and yes, it can have some benefit.  
But increase batch to fully use GPU resource might have better performance.

Thanks

---

<div class="post-metadata">

**Author:** ![CAT](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@CAT](https://forums.developer.nvidia.com/u/CAT)\
**Post date:** [May 18, 2020, 6:46am UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247/5 "2020-05-18T06:46:35Z")

</div>

Thank you.  
I was just a little confused while batching the input images.  
I am experiencing almost no benefit regarding inference time while batching.  
For example; When I use batchsize 1 for just a single image my network (A pretty big CNN) will take about 5ms,  
In contrast to that, when I use batchsize 4 it will take about 17ms. I was under the impression that batching 4 images wouldn’t take that much longer than inference on a single image.

---

<div class="post-metadata">

**Author:** ![1611074199](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@1611074199](https://forums.developer.nvidia.com/u/1611074199)\
**Post date:** [June 8, 2021, 3:08am UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247/6 "2021-06-08T03:08:43Z")

</div>

Hello, I want to know how can it run concurrently

---

<div class="post-metadata">

**Author:** ![RezaSa](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@RezaSa](https://forums.developer.nvidia.com/u/RezaSa)\
**Post date:** [February 14, 2022, 2:48pm UTC](https://forums.developer.nvidia.com/t/multiple-concurrent-execution-contexts/123247/9 "2022-02-14T14:48:34Z")

</div>

> [@CAT](#):
>
> batch-inference in a single IExecutionContext

Hi, CAT.  
I’m trying to get batch\_inferences from my engine, and I am wondering how I can do it.  
I’m having a CNN model for object detection, so my input size is fixed, and I was wondering how I could augment my GPU 's workload in any ways possible.  
Thanks.
