Can I Convert ONNX to TensorRT Engine with trtexec on a Machine Without GPU? Is the Output Engine Bound to the GPU Model?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi all,

I have two questions regarding converting an ONNX model to a TensorRT engine file using trtexec:

  1. Is it possible to perform this conversion on a machine that does not have an NVIDIA GPU installed, or is a GPU strictly required for the process?

  2. If a GPU is required (say, an RTX 4090 Ti), will the generated engine file be only compatible with that exact GPU model, or can it also run on other GPUs? For example, is there any hardware binding, or can the engine file be deployed across various GPUs (with possibly minor compatibility considerations)?

Thanks for your help!

4

Is it possible to perform this conversion on a machine that does not have an NVIDIA GPU installed, or is a GPU strictly required for the process?

There is a TensorRT product called TensorRT-RTX which allows an engine to be created without a GPU. When using traditional TensorRT Enterprise, then it’s not possible without a GPU of the same CUDA Compute Capability.

If a GPU is required (say, an RTX 4090 Ti), will the generated engine file be only compatible with that exact GPU model, or can it also run on other GPUs? For example, is there any hardware binding, or can the engine file be deployed across various GPUs (with possibly minor compatibility considerations)?

The engine file will be compatible with that specific TensorRT version and a GPU of the same family (same Compute Capability). A version or hardware compatible engine can be created to allow running on other TensorRT versions or other GPUs. This may have some performance loss. Some details on this topic can be found at Advanced Topics — NVIDIA TensorRT Documentation .