I want to use the TensorRT from ONNX file

I am using DGX.

Graphics: Tesla V100-DGXS-32GB/PCle/SSE2
Processor: Intel Xeon(R) CPU E5-2698 v4 @ 2.20GHz x 40
GNOME: 3.28.2
OS type: 64-bit
OS: Ubuntu 18.04.5 LTS

I want to convert Engine to ONNX to use TensorRT.

Which Docker container should I use?

I want to use the version, Tensorflow: 2.3 and Python:3.6

Should I use this container?

Thank you for your advice.

Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet


import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging

Sorry, I made a typing mistake and the display went wrong.
Re-write the script.

import sys, os, glob
import numpy as np
import tensorrt as trt
import common

from tensorflow.keras.preprocessing.image import img_to_array, load_img

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

def build_engine_onnx(model_file, batch_size):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(common.EXPLICIT_BATCH) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.max_workspace_size = common.GiB(1)
        builder.fp16_mode = True

    with open(model_file, 'rb') as model:
        if not parser.parse(model.read()):
            print('ERROR: Failed to parse the ONNX file.')
            for error in range(parser.num_errors):
            return None
    return builder.build_cuda_engine(network)

def main():
    model_file = ‘/home/--/sample.onnx’
    hold_vector = np.load(’/home/--/sample-hv.npy’)

    batch_size = 32
    with build_engine_onnx(model_file, batch_size) as engine, create_execution_context() as context:
        serialized_engine = engine.serialize()
        with open('samplename.engine', 'wb') as f:

if __name__ == "__main__":

and Then,

[TensorRT] ERROR: Network has dynamic or shape inputs, but no optimization profile has been defined.
[TensorRT] ERROR: Network validation failed.
Traceback (most recent call last):
with build_engine_onnex(model_file, batch_size) as engine, engine.create_executioncontext() as contexit:
AttributeError: __enter__

When I use the same script with Jetson NANO, I don’t get an error, so I think there is a problem with the Docker environment.

I want to use Docker to start the script, so it would be helpful if you could give me some advice on building the environment.

Thank you for your cooperation.

Hi @yoshitaka_tanaka_aa,

We suggest you to use Tensorflow NGC container.

Please refer TensorFlow Release Notes :: NVIDIA Deep Learning Frameworks Documentation, to choose latest container version matches to required Tensorflow, Python version.

For your reference, TensorRT installation guide,

Thank you.

1 Like

Hi, @spolisetty
Thank you for all the advice.
I am now able to avoid the error.
Thank you very much for your quick response.

1 Like