Converting FCN8-ResNet18 from Pytorch to TensorRT for inference on Jetson Nano

Description

Hi,

I am working on a project in which I trained a FCN8-ResNet18 model (thanks to this repository) using PyTorch.
Now I want to run my model on a Jetson Nano and I would like to optimize performance as mentionned in @dusty_nv’s article (here) thus I want to convert my ONNX model to TensorRT.

I used this repository to convert my model.
But it didn’t work.

The code used to convert my PyTorch model to ONNX locally :

# Export the model
torch.onnx.export(model_loaded,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "fcn8_resnet18.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable lenght axes
                                'output' : {0 : 'batch_size'}})

The command used to convert my ONNX model to TensorRT on the Jetson Nano :

onnx2trt fcn8_resnet18.onnx -o fc8_resnet18.trt

The output :

----------------------------------------------------------------
Input filename:   fcn8_resnet18.onnx
ONNX IR version:  0.0.6
Opset version:    10
Producer name:    pytorch
Producer version: 1.6
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
Parsing model
[2020-10-19 09:22:51 WARNING] [TRT]/home/lixo/background_substractor/onnx-tensorrt/onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
Building TensorRT engine, FP16 available:1
    Max batch size:     32
    Max workspace size: 1024 MiB
[2020-10-19 09:22:51   ERROR] Network has dynamic or shape inputs, but no optimization profile has been defined.
[2020-10-19 09:22:51   ERROR] Network validation failed.
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to create object
Aborted (core dumped)

I have also tried this python script on Jetson Nano :

def build_engine(onnx_file_path):
    # initialize TensorRT engine and parse ONNX model
    builder = trt.Builder(TRT_LOGGER)
    network = builder.create_network()
    parser = trt.OnnxParser(network, TRT_LOGGER)

    # parse ONNX
    with open(onnx_file_path, 'rb') as model:
        print('Beginning ONNX file parsing')
        parser.parse(model.read())
    print('Completed parsing of ONNX file')
    # allow TensorRT to use up to 1GB of GPU memory for tactic selection
    builder.max_workspace_size = 1 << 30
    # we have only one image in batch
    builder.max_batch_size = 34
    # use FP16 mode if possible
    if builder.platform_has_fast_fp16:
        builder.fp16_mode = True
    # generate TensorRT engine optimized for the target platform
    print('Building an engine...')
    last_layer = network.get_layer(network.num_layers - 1)
    if not last_layer:
        network.mark_output(last_layer.get_output(0))
    engine = builder.build_cuda_engine(network)
    #context = engine.create_execution_context()
    print("Completed creating Engine")

    return engine

def main():
    # initialize TensorRT engine and parse ONNX model
    ONNX_FILE_PATH = "fcn8_resnet18.onnx"
    engine = build_engine(ONNX_FILE_PATH)

    engine_file_path = 'fcn8_resnet18.trt'
    with open(engine_file_path, "wb") as f:
        f.write(engine.serialize())

if __name__ == "__main__":
    # execute only if run as a script
    main()

And I got the following output :

Beginning ONNX file parsing
Completed parsing of ONNX file
Building an engine...
python3: ../builder/Network.cpp:1187: virtual nvinfer1::ILayer* nvinfer1::Network::getLayer(int) const: Assertion `layerIndex >= 0' failed.
Aborted (core dumped) 

Thanks for you help.

Environment

On a Jetson Nano
TensorRT Version: 7.1.3.0
Torchvision Version: 0.7.0a0+78ed10c
Python Version (if applicable): 3.6.9
PyTorch Version (if applicable): 1.6.0

Relevant Files

my ONNX model : [here]

Hi @camille,

Please refer to below link for dynamic shape model:


You can also try trtexec command: (Need to use explicitBatch flag as well)

Thanks

Hi @SunilJB
Thank you for your response.
I solved my problem using trtexec
Have a good day