Running on multiple compute capabilities with one TRT Engine

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi, we’d like to run TensorRT 6 or 8 on both Jetson AGX and in AWS (Turing T4 on Arm). Do we have to build different engines locally on each platform or is there a way to generate PTX or at least force TRT to generate multiple CCs from one platform run?

Thank you

2

Hi,
This looks like a Jetson issue. Please refer to the below samlples in case useful.

For any further assistance, we recommend you to raise it to the respective platform from the below link

Thanks!

4

Thank you for your reply. This is not a Jetson issue, it’s a TensorRT issue.

5

This is not a Jetson-specific issue.
Is there any way TensorRT can be enabled to generate support for multiple CCs or PTX in the same engine file?

6

Tensorrt doesn’t support that. Yes, we have to build different engines locally on each platform
The generated plan/engine files are not portable across platforms or TensorRT versions. Plans are specific to the exact GPU model they were built on (in addition to the platforms and the TensorRT version) and must be re-targeted to the specific GPU.

Thanks!

7

Thank you for confirming. You guys may want to add support for PTX/alternate path/or cuDNN fallback for use cases when performance is not important but let’s say the same code needs to run in CI pipeline on a card with a different CC.

1 Like
8

Because the cost of switching versions/different CC is sometimes stratospheric. I spent the last 120 hrs trying to get running on CC 8.6 the model that was already previously running and exporting fine on Turing. Can’t say i’m a huge fan of TensorRT at the moment.