tensorflow to tensorRT - jupyter kernel keeps crashing

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I am trying to follow the instructions here: https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#import_tf_python

I have converted my frozen tensorflow graph (pre-trained keras mobilenetv2) to uff using:

$ python /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py -o model.uff frozen_graph.pb

It outputs the information about the input and output nodes. I copied those names down and use them in the builder code.

In my jupyter notebook I have the following code:

import tensorrt as trt
model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", (-1, 224, 224, 3))
    parser.register_output("Logits/Softmax")
parser.parse(model_file, network)

However, whenever I try and run this code my kernel restarts. It won’t ever run. Any ideas?

My pb file, uff file, and notebook file can be found here: GitHub - stonepreston/tf-to-trt-help

Also, why arent the tensorRT samples available on the nano? They arent in /usr/lib/python3.6/dist-packages/tensorrt? Where can I find them?

Edit: So i moved everything under the with and added a few more lines to construct the engine. However it looks like the engine is not being constructed correctly.

import tensorrt as trt
model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", (-1, 224, 224, 3))
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)
    builder.max_batch_size = max_batch_size
    builder.max_workspace_size = 1 <<  20 
    engine = builder.build_cuda_engine(network)
    with open(“sample.engine”, “wb”) as f:
		f.write(engine.serialize())

I get the error ‘NoneType’ objects has no attribute ‘serialize’ with the engine object.

2

Hi,

We have tested your model. It looks like the file name doesn’t match.
uff_parser return null since there is no model named as “model.uff”.

After renaming the model, there are still some errors related to the inappropriate layer name.
Please help to re-check the correct input/output name of model first.

Thanks.

3

Yeah that was my bad. I changed the name of the uff file before I uploaded to github. Ive updated the code to reflect the correct model name. I got the name of the inputs and outputs from the output of the convert_to_uff.py script.

$ python /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py -o model.uff frozen_graph.pb

The inputs were given as:

=== Automatically deduced input nodes ===
[name: "input_2"
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: -1
      }
      dim {
        size: 224
      }
      dim {
        size: 224
      }
      dim {
        size: 3
      }
    }
  }
}
]
=========================================

The outputs were given as:

=== Automatically deduced output nodes ===
[name: "Logits/Softmax"
op: "Softmax"
input: "Logits/BiasAdd"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}
]
==========================================

I verified these input and output names in tensorboard and they are correct.

the parser.parse method is returning false, indicating something is still going wrong at that step. Is there a way to log the errors that are occurring with the parser?

import tensorrt as trt
model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", (-1, 224, 224, 3))
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)
    builder.max_batch_size = 1
    builder.max_workspace_size = 1 <<  20 
    engine = builder.build_cuda_engine(network)
    with open("sample.engine", "wb") as f:
        f.write(engine.serialize())

The updated code and updated files are in the same repo (GitHub - stonepreston/tf-to-trt-help)

4

Hi,

Sorry for the late update.

There are some typo inside your implementation.
Please use for the dimension argument instead of ().

Like this:

--- parser.register_input("input_2", <b>(</b>-1, 224, 224, 3<b>)</b>)
+++ parser.register_input("input_2", <b>[</b>224, 224, 3<b>]</b>)

Thanks.

5

Whoops, yeah definitely needs to be brackets. Alright I tried that. Im still getting the same NoneType has no attribute serialize error. Parser still isnt parsing it correctly it seems.

import tensorrt as trt

model_file = 'model.uff'

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", [224, 224, 3])
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)
    builder.max_batch_size = 1
    builder.max_workspace_size = 1 <<  20 
    engine = builder.build_cuda_engine(network)
    with open("sample.engine", "wb") as f:
        f.write(engine.serialize())
6

Hi,

I can build your model without issue.
Here is my script for your reference:

python3 /usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py -o model.uff frozen_model.pb

import tensorrt as trt

max_batch_size = 1
model_file = 'model.uff'
TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser:
    parser.register_input("input_2", [3, 224, 224])
    parser.register_output("Logits/Softmax")
    parser.parse(model_file, network)

    builder.max_batch_size = max_batch_size
    builder.max_workspace_size = 1 <<  20

    engine = builder.build_cuda_engine(network)
    with open("sample.engine", "wb") as f:
        f.write(engine.serialize())

Thanks.

7

that worked! thanks. Guess I just needed to flip the input dimensions.

10

My Jupyter notebook and Colab kernel getting crashed while converting Tensorflow model to Tensor RT
Can anyone Please resolve this problem

#model path
input_saved_model_dir="/home/raguhtic/Codes/Tensorflow_to_Tenor_RT testing/Models/fast_depth_full5"

#defining function for tensorflow to tensor rt conversion
def convert_to_trt_graph_and_save(precision_mode="float32",input_saved_model_dir="model_path",calibration_data=batched_input):

    #SPECIFING THE CONVERSION USING IF STATEMENTS
    #if we want to convert into float 32 model
    if precision_mode=="float32":
        #converting precision to float 32
        precision_mode=trt.TrtPrecisionMode.FP32
        #adding a new path for tensor rt model
        coverted_save_suffix="_TFTRT_FP32"

    # if we want to convert into float 16 model
    if precision_mode == "float16":
        # converting precision to float 16
        precision_mode = trt.TrtPrecisionMode.FP16
        # adding a new path for tensor rt model
        coverted_save_suffix = "_TFTRT_FP16"

    # if we want to convert into int 8 model
    if precision_mode == "int8":
        # converting precision to int 8
        precision_mode = trt.TrtPrecisionMode.INT8
        # adding a new path for tensor rt model
        coverted_save_suffix = "_TFTRT_INT8"

    output_saved_model_dir = input_saved_model_dir+coverted_save_suffix


    #SPECIFY THE CONVERSION ACCORDING TO THE IF STATEMENTS GIVEN ABOVE


    conversion_params =trt.DEFAULT_TRT_CONVERSION_PARAMS._replace(

        precision_mode=precision_mode,#specifing the precision mode
        # max_workspace_size_bytes=3000000000 ,#gpu allocation memeory to procedss tensor rt(1GB)
        use_calibration=True
    )


    #TO TRT GRAPH CONVERTER OBJECT
    converter=trt.TrtGraphConverterV2(
       input_saved_model_dir=input_saved_model_dir,#giving the model diretory as input
        conversion_params=conversion_params#specifing the conversion parmas we gave
     )

    print("converting"+input_saved_model_dir+"model to"+coverted_save_suffix)

    #FOR INT 8 CONVERSION AN ADDITIONAL STEP HAS TO BE DONE BEFORE CONVERSION
    #A CALIBRATION INPUT FUNCTION IS GIVE TO FIND THE WEIGHTS AND BIAS RANGE AND FIT IT INTO INT 8 RANGE


    #convertig using Converter

    if precision_mode==trt.TrtPrecisionMode.INT8:
        def calibration_input_fn():
            yield (calibration_data,)

        converter.convert(calibration_input_fn=calibration_input_fn)

    else:
        converter.convert()

    converter.save(output_saved_model_dir)
    print("Conversion Completed")


#CALLING FUNCTION
conversion=convert_to_trt_graph_and_save("float16",input_saved_model_dir,batched_input)
11

Hi Ragu_23,

Please help to open a new topic with more details.

Thanks