I want to use the TensorRT from ONNX file

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi,
I am using DGX.

Graphics: Tesla V100-DGXS-32GB/PCle/SSE2
Processor: Intel Xeon(R) CPU E5-2698 v4 @ 2.20GHz x 40
GNOME: 3.28.2
OS type: 64-bit
OS: Ubuntu 18.04.5 LTS

I want to convert Engine to ONNX to use TensorRT.

Which Docker container should I use?

I want to use the version, Tensorflow: 2.3 and Python:3.6

Should I use this container?
→ nvcr.io/nvidia/tensorflow:19.09-py3

Thank you for your advice.

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec
In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

5

Sorry, I made a typing mistake and the display went wrong.
Re-write the script.

import sys, os, glob
import numpy as np
import tensorrt as trt
import common

from tensorflow.keras.preprocessing.image import img_to_array, load_img

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

def build_engine_onnx(model_file, batch_size):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(common.EXPLICIT_BATCH) as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.max_workspace_size = common.GiB(1)
        builder.fp16_mode = True

    with open(model_file, 'rb') as model:
        if not parser.parse(model.read()):
            print('ERROR: Failed to parse the ONNX file.')
            for error in range(parser.num_errors):
                print(parser.get_error(error))
            return None
    return builder.build_cuda_engine(network)

def main():
    model_file = ‘/home/--/sample.onnx’
    hold_vector = np.load(’/home/--/sample-hv.npy’)

    batch_size = 32
    with build_engine_onnx(model_file, batch_size) as engine, create_execution_context() as context:
        serialized_engine = engine.serialize()
        with open('samplename.engine', 'wb') as f:
            f.write(engine.serialize())

if __name__ == "__main__":
    main()

and Then,

[TensorRT] ERROR: Network has dynamic or shape inputs, but no optimization profile has been defined.
[TensorRT] ERROR: Network validation failed.
Traceback (most recent call last):
~~
with build_engine_onnex(model_file, batch_size) as engine, engine.create_executioncontext() as contexit:
AttributeError: __enter__

When I use the same script with Jetson NANO, I don’t get an error, so I think there is a problem with the Docker environment.

I want to use Docker to start the script, so it would be helpful if you could give me some advice on building the environment.

Thank you for your cooperation.

6

Hi @yoshitaka_tanaka_aa,

We suggest you to use Tensorflow NGC container.

Please refer TensorFlow Release Notes :: NVIDIA Deep Learning Frameworks Documentation, to choose latest container version matches to required Tensorflow, Python version.

For your reference, TensorRT installation guide,

Thank you.

1 Like
7

Hi, @spolisetty
Thank you for all the advice.
I am now able to avoid the error.
Thank you very much for your quick response.

1 Like