How can I convert onnx model to engine model supporting a GPU with different compute capability on another pc

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
1

I’m using a laptop to convert an onnx model to engine model, and then run the engine model on a gpu.
My laptop’s GPU is “NVIDIA GeForce RTX 3060 Laptop GPU“, which’s compute capability is 8.6.”
The engine model converted above can run on my laptop. But it can’t run on another PC，which‘s GPU is “NVIDIA GeForce GTX 1660 Ti” and compute capability is 7.5. The exception show “The engine plan file is generated on an incompatible device, expecting compute 7.5 gt compute 8.6.please rebuild.”

So I wonder on my laptop, how can I convert the onnx to engine model which support the PC (GPU 1660, compute capability 7.5). Hope anyone can give me some suggestions, thanks so much!

My loptop and PC has the same below configuration:

  1. Cuda 11.3
  2. Dudnn 6.14
  3. TensorRT 8.6.1

My python code convert onnx model to engine model is as below:


import torch
import onnx
import tensorrt as trt

onnx_model = 'model.onnx'

output_names=['output'], opset_version=11)
onnx_model = onnx.load(onnx_model)

logger = trt.Logger(trt.Logger.ERROR)
builder = trt.Builder(logger)

EXPLICIT_BATCH = 1 << (int)(
    trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
network = builder.create_network(EXPLICIT_BATCH)

parser = trt.OnnxParser(network, logger)

if not parser.parse(onnx_model.SerializeToString()):
    error_msgs = ''
    for error in range(parser.num_errors):
        error_msgs += f'{parser.get_error(error)}\n'
    raise RuntimeError(f'Failed to parse onnx, {error_msgs}')

config = builder.create_builder_config()
config.max_workspace_size = 1<<20
profile = builder.create_optimization_profile()

profile.set_shape('input', [1, 1, 1440, 1440], [1, 1, 1440, 1440], [1, 1, 1440, 1440])
config.add_optimization_profile(profile)

engine = builder.build_engine(network, config)

with open('model.engine', mode='wb') as f:
    f.write(bytearray(engine.serialize()))
    print("generating file done!")
2

Hello,

Thanks for posting your questions here, but this forum topic is about “Nsight Graphics”, but not for “TensorRT” or AI related topics. Maybe you can move this to another more specific topic area for getting more professional support.

Thanks
An

3

I went ahead and moved this topic to the TensorRT forum.

4

Sorry, I am not very familiar with this. Thanks for your post, and how can I get my new websit?

5

It is currently impossible to do so without building your engine on actual GPU of compute capacity 7.5 for TensorRT. TensorRT used to be strict hardware specific. TensorRT now does have a hardware compatible mode, documented here: Developer Guide :: NVIDIA Deep Learning TensorRT Documentation, but we only support Ampere+ GPU, which means, you can build a hardware-compatible engine on your 3060, and run on 3090, A100, H100, etc, but not your 1660Ti.

6

Thanks so much for your answer!

I wonder is there because they are not the same serise or not, or because it is only compatible with higher version.
Besides, Ampere+ GPU means?

Could you give more explanation please?

7

Ampere+ means Ampere, Ada Lovelace, Hopper, etc. that has compute capability of 8.0+. This is TensorRT’s design choice - we are not supporting 1660Ti as part of hardware compatible design at this moment, and I can’t share more details than the developer guide. Please follow the developer guide to figure out what you can play with.

8

OK，thanks very much！