tensorflow to tensorRT - jupyter kernel keeps crashing

I am trying to follow the instructions here: https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#import_tf_python

I have converted my frozen tensorflow graph (pre-trained keras mobilenetv2) to uff using:

$ python /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py -o model.uff frozen_graph.pb

It outputs the information about the input and output nodes. I copied those names down and use them in the builder code.

In my jupyter notebook I have the following code:

import tensorrt as trt
model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", (-1, 224, 224, 3))
    parser.register_output("Logits/Softmax")
parser.parse(model_file, network)

However, whenever I try and run this code my kernel restarts. It won’t ever run. Any ideas?

My pb file, uff file, and notebook file can be found here: https://github.com/stonepreston/tf-to-trt-help

Also, why arent the tensorRT samples available on the nano? They arent in /usr/lib/python3.6/dist-packages/tensorrt? Where can I find them?

Edit: So i moved everything under the with and added a few more lines to construct the engine. However it looks like the engine is not being constructed correctly.

import tensorrt as trt
model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", (-1, 224, 224, 3))
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)
    builder.max_batch_size = max_batch_size
    builder.max_workspace_size = 1 <<  20 
    engine = builder.build_cuda_engine(network)
    with open(“sample.engine”, “wb”) as f:
		f.write(engine.serialize())

I get the error ‘NoneType’ objects has no attribute ‘serialize’ with the engine object.

Hi,

We have tested your model. It looks like the file name doesn’t match.
uff_parser return null since there is no model named as “model.uff”.

After renaming the model, there are still some errors related to the inappropriate layer name.
Please help to re-check the correct input/output name of model first.

Thanks.

Yeah that was my bad. I changed the name of the uff file before I uploaded to github. Ive updated the code to reflect the correct model name. I got the name of the inputs and outputs from the output of the convert_to_uff.py script.

$ python /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py -o model.uff frozen_graph.pb

The inputs were given as:

=== Automatically deduced input nodes ===
[name: "input_2"
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: -1
      }
      dim {
        size: 224
      }
      dim {
        size: 224
      }
      dim {
        size: 3
      }
    }
  }
}
]
=========================================

The outputs were given as:

=== Automatically deduced output nodes ===
[name: "Logits/Softmax"
op: "Softmax"
input: "Logits/BiasAdd"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}
]
==========================================

I verified these input and output names in tensorboard and they are correct.

the parser.parse method is returning false, indicating something is still going wrong at that step. Is there a way to log the errors that are occurring with the parser?

import tensorrt as trt
model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", (-1, 224, 224, 3))
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)
    builder.max_batch_size = 1
    builder.max_workspace_size = 1 <<  20 
    engine = builder.build_cuda_engine(network)
    with open("sample.engine", "wb") as f:
        f.write(engine.serialize())

The updated code and updated files are in the same repo (https://github.com/stonepreston/tf-to-trt-help)

Hi,

Sorry for the late update.

There are some typo inside your implementation.
Please use for the dimension argument instead of ().

Like this:

--- parser.register_input("input_2", <b>(</b>-1, 224, 224, 3<b>)</b>)
+++ parser.register_input("input_2", <b>[</b>224, 224, 3<b>]</b>)

Thanks.

Whoops, yeah definitely needs to be brackets. Alright I tried that. Im still getting the same NoneType has no attribute serialize error. Parser still isnt parsing it correctly it seems.

import tensorrt as trt

model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", [224, 224, 3])
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)
    builder.max_batch_size = 1
    builder.max_workspace_size = 1 <<  20 
    engine = builder.build_cuda_engine(network)
    with open("sample.engine", "wb") as f:
        f.write(engine.serialize())

Hi,

I can build your model without issue.
Here is my script for your reference:

python3 /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py -o model.uff frozen_model.pb
import tensorrt as trt

max_batch_size = 1
model_file = 'model.uff'
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", [3, 224, 224])
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)

    builder.max_batch_size = max_batch_size
    builder.max_workspace_size = 1 <<  20

    engine = builder.build_cuda_engine(network)
    with open("sample.engine", "wb") as f:
        f.write(engine.serialize())

Thanks.

that worked! thanks. Guess I just needed to flip the input dimensions.