TensorRT7 IExecutionContext with dynamic shapes

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi!
We are just migrating our system from TensorRT5 to TensorRT7, to get the benefits of dynamic shapes.
Then we just realized the following statement in the documentation:

If the engine supports dynamic shapes, each execution context in concurrent use must use a separate optimization profile. (nvinfer1::IExecutionContext Class Reference)

The statement leads to the fact, that we have to know the number of IExecutionContexts at ICudaEngine creation time. So no real dynamic scaling can be done with IExecutionContext at runtime in TensorRT7. So this is some kind of feature loss, I think..

So my question are :
Are you planing to (re)support this feature in the near future?
What is the reason to force unique usage of OptimizationProfiles?

Thanks!

Environment

TensorRT Version:7.2.x
GPU Type: All
Nvidia Driver Version: 450.xx
CUDA Version: 10.2
CUDNN Version:8.0.x
Operating System + Version: Ubuntu 18.04

3

Hi @locki.janos.m
This is a known issue. It was there due to some potential race condition. And we are now working on separating resources that may have conflict.
Please stay tuned for further updates.

Thanks!

4

Hello!
Is there any news about this feature loss (re)support?

Thanks!

5

Hi,
Please refer to the installation steps from the below link if in case you are missing on anything

Also, we suggest you to use TRT NGC containers to avoid any system dependency related issues.

Thanks!

7

I hope I can get a real answer…