# \[Question\] trtexec understanding issue

**URL:** <https://forums.developer.nvidia.com/t/question-trtexec-understanding-issue/196370>\
**Category:** TensorRT\
**Created:** [November 26, 2021, 11:23pm UTC](https://forums.developer.nvidia.com/t/question-trtexec-understanding-issue/196370 "2021-11-26T23:23:48Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![redradist](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/redradist/32/49835_2.png) [@redradist](https://forums.developer.nvidia.com/u/redradist)\
**Post date:** [November 26, 2021, 11:23pm UTC](https://forums.developer.nvidia.com/t/question-trtexec-understanding-issue/196370/1 "2021-11-26T23:23:48Z")

</div>

Hi all,

Recently I’ve faced with an issue that when I run trtexec with option “–streams=14” only one CUDA stream executes all infers !!

But when I add option “–threads=14”, then 14 CUDA threads execute in parallel …

But why ? I saw that even in single thread there are used enqueueV2 API that should execute all infers in parallel !!  
But to see parallel execution i need to add this “–threads” option …

Are there some limitation of enqueueV2 ? Maybe it executes in single thread in some cases ??

---

<div class="post-metadata">

**Author:** ![redradist](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/redradist/32/49835_2.png) [@redradist](https://forums.developer.nvidia.com/u/redradist)\
**Post date:** [November 26, 2021, 11:43pm UTC](https://forums.developer.nvidia.com/t/question-trtexec-understanding-issue/196370/2 "2021-11-26T23:43:03Z")

</div>

Seems like it happens, because:

```cpp
    IterationStreams iStreams;
    for (int s = 0; s < streams; ++s)
    {
        Iteration* iteration = new Iteration(offset + s, inference, *iEnv.context[offset], *iEnv.bindings[offset]);
        ...
    }

```

the same context was used for multiple streams and according to the documentation for enqueueV2 it is undefined behavior:

```auto
Calling enqueueV2() in from the same IExecutionContext object with different CUDA streams concurrently results in undefined behavior. To perform inference concurrently in multiple streams, use one execution context per stream. 

```

Am I right ?

But then why do you have in TensorRT such bad example with undefined behavior ?

---

<div class="post-metadata">

**Author:** ![redradist](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/redradist/32/49835_2.png) [@redradist](https://forums.developer.nvidia.com/u/redradist)\
**Post date:** [November 27, 2021, 10:01am UTC](https://forums.developer.nvidia.com/t/question-trtexec-understanding-issue/196370/3 "2021-11-27T10:01:05Z")

</div>

Seems like I have found similar issue [multi-stream parallel execution with one GPU ERROR · Issue #846 · NVIDIA/TensorRT · GitHub](https://github.com/NVIDIA/TensorRT/issues/846#issuecomment-716403960)

---

<div class="post-metadata">

**Author:** ![NVES](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nves/32/14043_2.png) [@NVES](https://forums.developer.nvidia.com/u/NVES)\
**Post date:** [November 29, 2021, 9:40am UTC](https://forums.developer.nvidia.com/t/question-trtexec-understanding-issue/196370/4 "2021-11-29T09:40:04Z")

</div>

Hi,  
Please refer to the below link for Sample guide.

> **[Sample Support Guide :: NVIDIA Deep Learning TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/sample-support-guide/index.html)**
>
> This Samples Support Guide provides an overview of all the supported NVIDIA TensorRT 8.4.3 samples included on GitHub and in the product package. The TensorRT samples specifically help in areas such as recommenders, machine comprehension, character...

Refer to the installation steps from the link if in case you are missing on anything

> **[Installation Guide :: NVIDIA Deep Learning TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html)**
>
> This NVIDIA TensorRT 8.4.1 Installation Guide provides the installation requirements, a list of what is included in the TensorRT package, and step-by-step instructions for installing TensorRT.

However suggested approach is to use TRT NGC containers to avoid any system dependency related issues.

> **[TensorRT | NVIDIA NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorrt)**
>
> NVIDIA TensorRT is a C++ library that facilitates high-performance inference on NVIDIA graphics processing units (GPUs). TensorRT takes a trained network and produces a highly optimized runtime engine that performs inference for that network.

In order to run python sample, make sure TRT python packages are installed while using NGC container.  
/opt/tensorrt/python/python\_setup.sh

In case, if you are trying to run custom model, please share your model and script with us, so that we can assist you better.  
Thanks!

---

<div class="post-metadata">

**Author:** ![redradist](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/redradist/32/49835_2.png) [@redradist](https://forums.developer.nvidia.com/u/redradist)\
**Post date:** [December 6, 2021, 10:17am UTC](https://forums.developer.nvidia.com/t/question-trtexec-understanding-issue/196370/6 "2021-12-06T10:17:26Z")

</div>

Hi @NVES,

It is not related to custom model, it is related to undefined behaviour in you trtexec example as I described above …  
Running enqueueV2 on different streams is undefined behaviour, and you has such undefined behaviour in your code