TensorRT: parallel inference when most input shapes locate in the same optimization profile

sneaxiy · May 19, 2021, 1:31pm

I use TensorRT to do the multi-threaded online serving. My model is built with dynamic shape support. The optimization profile is like that: min_dim = [2^k + 1, other_shape], opt_dim = [2^k, other_shape], max_dim = [2^k, other_shape], where k is the profile index inside [0, N) and other_shape is a deterministic integer number.

After I read the doc of TensorRT dynamic shape engine, I found that:
(1) IExecutionContext is not thread-safe.
(2) Different IExecutionContext objects created from the same ICudaEngine object cannot share the same optimization profile index. That is to say, the optimization profile index must be different in different IExecutionContext objects.
(3) It is recommended that each IExecutionContext object should use a unique CUDA stream.

In conclusion, when I use the engine with dynamic shapes, I should:
(1) create N IExecutionContext objects.
(2) set the optimization profile to be 0,1,…N-1 for each IExecutionContext object.
(3) create N CUDA streams, and assign each of them to each IExecutionContext object.

My question is: if most of the requested input shapes are inside the same optimization profile, it means that these requests would be processed by a single IExecutionContext object and a single CUDA stream. These requests can only be processed one by one but not in parallel. How to solve this problem if I want more parallelism? Is there any best practice?

NVES · May 19, 2021, 1:38pm

Hi,
Can you try running your model with trtexec command, and share the “”–verbose"" log in case if the issue persist

You can refer below link for all the supported operators list, in case any operator is not supported you need to create a custom plugin to support that operation

github.com

onnx/onnx-tensorrt/blob/master/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 7.2 supports operators up to Opset 13. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT where possible. If not possible, TensorRT will throw an error. See the [TensorRT layer support matrix](https://docs.nvidia.com/deeplearning/sdk/tensorrt-support-matrix/index.html#layers-precision-matrix) for more information on data type support.

## Operator Support Matrix

| Operator                  | Supported? | Restrictions                                                                                                                             |
|---------------------------|------------|------------------------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          |
| Acos                      | Y          |
| Acosh                     | Y          |
| Add                       | Y          |
| And                       | Y          |
| ArgMax                    | Y          |

This file has been truncated. show original

Also, request you to share your model and script if not shared already so that we can help you better.

Thanks!

sneaxiy · May 19, 2021, 1:44pm

@NVES I am sorry that the docs you recommended do not solve my problem. Could you please take a look in detail?

spolisetty · May 24, 2021, 9:33am

Hi @sneaxiy,

Your understanding of execution context is correct. For now, we suggest you to create multiple profiles with same size or have multiple engine instances.

Thank you.

sneaxiy · May 24, 2021, 1:23pm

Thanks! As you have suggested, I can create multiple engine instances. Is there any method that I can share the weights between these different ICudaEngine objects?

spolisetty · May 25, 2021, 5:35pm

Hi @sneaxiy,

You can create single engine with multiple contexts. Weights would be shared among the contexts.

Thank you.

sneaxiy · June 11, 2021, 3:57pm

Thank you for your reply. I still have one question: if I deserialize a same engine file to create multiple ICudaEngine instances, is there any method that I can share the weights between these different ICudaEngine instances? Since these instances are deserialized from the same engine file, they can share the same weights so that I can save the GPU memory consumption.

spolisetty · June 18, 2021, 1:01pm

Hi @sneaxiy,

Looks like there is no ideal solution for this as of now. We recommend you to try create multiple profile, and assign each profile to one of the instance instead of deserialize multiple cuda engines.
https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#opt_profiles