TensorRT graph to slow

Hi there

I trained a graph with ssd_inception an converted it to a rt-graph to get faster results.
But the code is still way to slow, almost stocking…

Could need some help, here is my Implementation:
The Programm is for detecting a drone an calculating the distance.

import cv2
import numpy as np
import os
import six.moves.urllib as urllib
import sys
import tarfile
import tensorflow as tf
import tensorflow.contrib.tensorrt as trt
#tf.contrib.resampler
import zipfile
from collections import defaultdict
from io import StringIO
from matplotlib import pyplot as plt
from PIL import Image
#from win32api import GetSystemMetrics
import glob
import subprocess

This is needed since the notebook is stored in the object_detection folder.

sys.path.append("…")

from utils import label_map_util
from utils import visualization_utils as vis_util

cap1 = cv2.VideoCapture(1)
cap2 = cv2.VideoCapture(0)

CAMERA_WIDTH = 1920
CAMERA_HEIGHT = 1080

IMAGE_WIDTH = 640
IMAGE_HEIGHT = 480

#User set camera
cap1.set(3, CAMERA_WIDTH) # Set resolution width
cap1.set(4, CAMERA_HEIGHT) # Set resolution height
#retl_e = img_l.set(15, exposure) # Set exposure.
cap2.set(3, CAMERA_WIDTH) # Set resolution width
cap2.set(4, CAMERA_HEIGHT) # Set resolution hight
#drone image:
#droneIMG = cv2.resize(cv2.imread(‘backgroundIMG/drone_image.jpg’), (50,50))

output = subprocess.Popen(‘xrandr | grep “*” | cut -d" " -f4’,shell=True, stdout=subprocess.PIPE).communicate()[0]
screenX, screenY = [int(i) for i in output.decode(“utf-8”).replace("\n", “”).split(‘x’)]

for SIFT

MIN_MATCH_COUNT = 1

What model to download.

MODEL_NAME = ‘drone_graph’

Path to frozen detection graph. This is the actual model that is used for the object detection.

PATH_TO_CKPT = MODEL_NAME + ‘/TensorRT_model.pb’

List of the strings that is used to add correct label for each box.

PATH_TO_LABELS = os.path.join(‘training’, ‘object-detection.pbtxt’)

NUM_CLASSES = 1

print(“Preparing TensorFlow…”)
detection_graph = tf.Graph()
with detection_graph.as_default():
od_graph_def = tf.GraphDef()
with tf.gfile.GFile(PATH_TO_CKPT, ‘rb’) as fid:
serialized_graph = fid.read()
od_graph_def.ParseFromString(serialized_graph)
tf.import_graph_def(od_graph_def, name=’’)
tf_config = tf.ConfigProto()#Hannes memory laufend dazu nehmen
tf_config.gpu_options.allow_growth = True
‘’’
for op in detection_graph.get_operations():
print(op.name)
#exit()
‘’’
label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
category_index = label_map_util.create_category_index(categories)#Hannes

#calibrate cameras
REMAP_INTERPOLATION = cv2.INTER_LINEAR
#read parameters

rootDir = ‘./’
outputFile = rootDir + ‘output/calibration.npz’

calibration = np.load(outputFile, allow_pickle=False)
imageSize = tuple(calibration[“imageSize”])
leftMapX = calibration[“leftMapX”]
leftMapY = calibration[“leftMapY”]
leftROI = tuple(calibration[“leftROI”])
rightMapX = calibration[“rightMapX”]
rightMapY = calibration[“rightMapY”]
rightROI = tuple(calibration[“rightROI”])

R = calibration[“rotationMatrix”]
T = calibration[“translationVector”]
P1 = calibration[“leftProjection”]
P2 = calibration[“rightProjection”]

def load_image_into_numpy_array(image):
(im_width, im_height) = image.size
return np.array(image.getdata()).reshape(
(im_height, im_width, 3)).astype(np.uint8)

def compare_SIFT(box1, img1, box2, img2):
“”“code is in SIFTry, right now not working with live feed, so returning center of both boxes”""
pass

print(“Ready!”)
with detection_graph.as_default():
with tf.Session(graph=detection_graph) as sess:
while True:
retL,imageL = cap1.read()
retR,imageR = cap2.read()

  imageL = cv2.resize(imageL, (IMAGE_WIDTH, IMAGE_HEIGHT))
  imageR = cv2.resize(imageR, (IMAGE_WIDTH, IMAGE_HEIGHT))
  #whiteIMG = cv2.resize(cv2.imread('backgroundIMG/white.jpg'), (int(screenX/2),screenY))

  # for Left camera:
  
  # Expand dimensions since the model expects images to have shape: [1, None, None, 3]
  image_np_expandedL = np.expand_dims(imageL, axis=0)
  image_tensorL = detection_graph.get_tensor_by_name('image_tensor:0')
  # Each box represents a part of the image where a particular object was detected.
  boxesL = detection_graph.get_tensor_by_name('detection_boxes:0')
  
  # Each score represent how level of confidence for each of the objects.
  # Score is shown on the result image, together with the class label.
  scoresL = detection_graph.get_tensor_by_name('detection_scores:0')
  classesL = detection_graph.get_tensor_by_name('detection_classes:0')
  num_detectionsL = detection_graph.get_tensor_by_name('num_detections:0')
  
  # Actual detection.
  (boxesL, scoresL, classesL, num_detectionsL) = sess.run([boxesL, scoresL, classesL, num_detectionsL],feed_dict={image_tensorL: image_np_expandedL})

  # Visualization of the results of a detection.
  boxL = vis_util.visualize_boxes_and_labels_on_image_array(imageL, np.squeeze(boxesL), np.squeeze(classesL).astype(np.int32),
      np.squeeze(scoresL),
      category_index,
      use_normalized_coordinates=True,
      line_thickness=1)
  

  # for right camera:
  
  # Expand dimensions since the model expects images to have shape: [1, None, None, 3]
  image_np_expandedR = np.expand_dims(imageR, axis=0)
  image_tensorR = detection_graph.get_tensor_by_name('image_tensor:0')#image_tensor:0')
  
  # Each box represents a part of the image where a particular object was detected.
  boxesR = detection_graph.get_tensor_by_name('detection_boxes:0')
  
  # Each score represent how level of confidence for each of the objects.
  # Score is shown on the result image, together with the class label.
  scoresR = detection_graph.get_tensor_by_name('detection_scores:0')
  classesR = detection_graph.get_tensor_by_name('detection_classes:0')
  num_detectionsR = detection_graph.get_tensor_by_name('num_detections:0')
  
  # Actual detection.
  (boxesR, scoresR, classesR, num_detectionsR) = sess.run([boxesR, scoresR, classesR, num_detectionsR],feed_dict={image_tensorR: image_np_expandedR})

  # Visualization of the results of a detection.
  boxR = vis_util.visualize_boxes_and_labels_on_image_array(imageR, np.squeeze(boxesR), np.squeeze(classesR).astype(np.int32),
      np.squeeze(scoresR),
      category_index,
      use_normalized_coordinates=True,
      line_thickness=1)

  
  if boxL!=None and boxR!=None:
    # un-normalize box(integer values):
    im_widthL, im_heightL = Image.fromarray(np.uint8(imageL)).convert('RGB').size
    im_widthR, im_heightR = Image.fromarray(np.uint8(imageR)).convert('RGB').size
    
    yminL, xminL, ymaxL, xmaxL = boxL
    boxL = (xminL * im_widthL, xmaxL * im_widthL, yminL * im_heightL, ymaxL * im_heightL)
    yminL, xminL, ymaxL, xmaxL = boxL

    yminR, xminR, ymaxR, xmaxR = boxR
    boxR = (xminR * im_widthR, xmaxR * im_widthR, yminR * im_heightR, ymaxR * im_heightR)
    yminR, xminR, ymaxR, xmaxR = boxR
    
    #find same dots in both cameras:
    #just center of two boxes (problem with SIFT)
    #compare_SIFT(box1,image_np, box2, image_np2)
    centerL = ((yminL+xminL)/2, (ymaxL+xmaxL)/2)
    centerR = ((yminR+xminR)/2, (ymaxR+xmaxR)/2)
    
    #trangulate:
    arr = np.array([])
    arr = cv2.triangulatePoints(P1, P2, centerL, centerR, arr)
    arrCar = arr/arr[3]
    arrCar = np.delete(arrCar, (3), axis=0)
    print(centerL)
    print(centerR)
    print(arrCar)       

    cv2.circle(imageR, (int(centerR[0]), int(centerR[1])), 5, (0,0,255),5)#5, (0,0,255),10)
    cv2.circle(imageL, (int(centerL[0]), int(centerL[1])), 5, (0,0,255),5)
    
    cv2.imshow('right', imageL)
    cv2.imshow('left', imageR)

    #tranX = int(centerR[0] + Pr[0]) -25
    #tranY = int(centerR[1] + Pr[1]) -25 +120
    #whiteIMG[ tranY:50+tranY , tranX:50+tranX] = droneIMG
    #cv2.circle(whiteIMG, (320,360), 5, (0,0,0),10)
    #cv2.line(whiteIMG, (320,360),(tranX+25,tranY+25), (0,0,255),5)
    #whiteIMG = cv2.flip(whiteIMG,1)
    #distirng = "distance: " + "{}".format(round(Pr[2]/300,2))
    #cv2.putText(whiteIMG,distirng, (20,80), cv2.FONT_HERSHEY_SIMPLEX, 1, 255)
    #cv2.imshow('world', whiteIMG)
    
  if cv2.waitKey(25) & 0xFF == ord('q'):
    cv2.destroyAllWindows()
    print("Done!")
    break

Thanks.

Hi,

Can you provide the following information so we can better help?
Provide details on the platforms you are using:
o Linux distro and version
o GPU type
o Nvidia driver version
o CUDA version
o CUDNN version
o Python version [if using python]
o Tensorflow and PyTorch version
o TensorRT version

Also, if possible please share the script & model file to reproduce the issue.

Thanks

-jetson tx2/Jetpack 4.2 with ubuntu 18.04
-CUDA 10
-python 3.6
-TensorRT 5.0.6
-cuDNN 7.3.1
-tensorflow 1.13.1

Hi,
In above code snippet I am not able to find the trt-graph conversion process.
Could you please share the script and model file as well to reproduce the issue?

Meanwhile, could you please try latest Jetpack version?
https://developer.nvidia.com/embedded/jetpack

Please refer to below sample examples:
https://github.com/NVIDIA-AI-IOT/tf_trt_models/tree/master/examples

Thanks

sry, here you can find how I converted to the trt-graph.
tr_trt.zip (1.22 KB)

Can no one help me?

I don’t know what to try anymore…

If I start the python file it takes like 20min till the cameras show up. And then its around 4-5fps… It should be way faster for real-time detection.

What I am doing wrong?

Hi,

As requested earlier, could you please share the model file as well so we can better help?
Meanwhile, please refer to below sample:
https://github.com/NVIDIA-AI-IOT/tf_trt_models/blob/master/examples/detection/detection.ipynb

Few queries:

  1. How many ops are converted to TensorRT engine?
  2. What’s the performance of model before optimization?

Thanks

I am sorry seems like the files did not upload correctly.

Here are again the files in a zip.
Its the frozen graph, the TensorRT model after the transformation and the file test3.py with which I tested it.

The performance before optimization is even worse. Its getting stuck and is not able to calculate the distances properly.
test.zip (82.7 MB)

And I took a look at the example and I am not sure how or where to use them in my code.

I my earlier post you can see the tr_trt file which I used.

Hi,

It seems that your input shapes are not fully defined. Due to which TRT engines are falling back to TF during conversion.
image_tensor --> uint8[?,?,?,3]
After conversion:
numb. of trt_engine_nodes in TensorRT graph: 0

With “is_dynamic_op=True” flag i was able to get numb. of trt_engine_nodes in TensorRT graph: 5
Could you please try “is_dynamic_op” or make sure all the input shapes are defined in your model?
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#tf-trt-api

Another alternative is to convert your model to ONNX instead using tf2onnx and then convert to TensorRT using ONNX parser. Any layer that are not supported needs to be replaced by custom plugin to get better performance.
https://github.com/onnx/tensorflow-onnx
https://github.com/onnx/onnx-tensorrt/blob/master/operators.md

Thanks

Hi
Thank you.

I changed the is_dynamic_op parameter to False and was also able to get numb.of trt_engine_nodes 5.

Unfortunately when I again try my test.py for object detection and distance measurement it takes forever.

It is still running and I started the file 45min ago.

Here are the current outputs:

ivanflurina@nvidia:~/aktuell/Drone_Detection_RT$ python3 test3.py
Preparing TensorFlow…
Ready!
2020-01-02 13:42:49.634213: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-01-02 13:42:49.635146: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x8c3feb0 executing computations on platform Host. Devices:
2020-01-02 13:42:49.635224: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): ,
2020-01-02 13:42:49.748808: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:965] ARM64 does not support NUMA - returning NUMA node zero
2020-01-02 13:42:49.749320: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x8ba10b0 executing computations on platform CUDA. Devices:
2020-01-02 13:42:49.749390: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): NVIDIA Tegra X2, Compute Capability 6.2
2020-01-02 13:42:49.749881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.02
pciBusID: 0000:00:00.0
totalMemory: 7.67GiB freeMemory: 2.43GiB
2020-01-02 13:42:49.749956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-01-02 13:42:51.924794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-02 13:42:51.924915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2020-01-02 13:42:51.924966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2020-01-02 13:42:51.925239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1922 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2020-01-02 13:42:55.624685: I tensorflow/contrib/tensorrt/kernels/trt_engine_op.cc:496] Building a new TensorRT engine for TRTEngineOp_0 with batch size 1
2020-01-02 13:42:55.694790: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.695080: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.697395: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.697827: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.698207: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.699035: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.699424: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.700577: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.702295: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.702592: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.703090: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.703547: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.704724: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.705190: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.706330: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.708619: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.710005: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.711979: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.719690: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.720803: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.722245: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.724798: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.729437: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.730965: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.732588: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.734441: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.740425: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.746398: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.749739: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.755363: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.758235: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.763344: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.765508: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.770799: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.776478: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.779301: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.782672: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.785550: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.791944: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.794596: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.800630: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.809733: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.811980: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.816837: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.819860: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.828257: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.831410: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.842788: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.855537: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.857746: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.860978: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.869300: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.873344: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.891055: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.913083: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.926383: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.933317: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.953386: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.958864: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.971212: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.988925: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:55.993583: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.006722: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.013371: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.033857: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.041085: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.055493: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.072069: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.076881: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.087420: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.147741: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.155754: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.166327: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.171239: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.181925: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.186793: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:42:56.190575: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Tensor DataType is determined at build time for tensors not marked as input or output.
2020-01-02 13:46:00.095954: I tensorflow/contrib/tensorrt/kernels/trt_engine_op.cc:496] Building a new TensorRT engine for TRTEngineOp_3 with batch size 1
2020-01-02 13:46:00.095966: I tensorflow/contrib/tensorrt/kernels/trt_engine_op.cc:496] Building a new TensorRT engine for TRTEngineOp_2 with batch size 1
2020-01-02 13:46:00.334842: I tensorflow/contrib/tensorrt/kernels/trt_engine_op.cc:496] Building a new TensorRT engine for Postprocessor/TRTEngineOp_1 with batch size 1

Why does it take forever? Can you help me again?

Thanks so much

Hi,

In the dynamic mode, the TensorRT engines are built during runtime when the actual inference happens.
Since the engine will have to be built at this time, this first batch will take longer to execute than usual. If you later send more images with the same shape, the previously built engine will be used immediately with no additional overhead.

Could you please try updating the Tensorflow/TRT version and use the below API for model conversion?
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#using-savedmodel

The original Python function create_inference_graph that was used in TensorFlow 1.13 and earlier is deprecated in TensorFlow >1.13 and removed in TensorFlow 2.0.

Also, please refer below link to optimize the performance:
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#performance

Another alternative is to convert your model to ONNX instead using tf2onnx and then convert to TensorRT using ONNX parser. Any layer that are not supported needs to be replaced by custom plugin to get better performance.
https://github.com/onnx/tensorflow-onnx
https://github.com/onnx/onnx-tensorrt/blob/master/operators.md

Thanks

Hello
I tried to upgrade TF with
pip3 install tensorflow == 1.14

and I get the following error:

ERROR: Could not find a version that satisfies the requiremenr tensorflow == 1.14
(from versions: none)
ERROR: No matchibg distribution found for tensorflow == 1.14

How can I resolve this?

I shared how I trained a custom SSD object (hand) detector with tensorflow object detection API, and how I optimized the trained hand detection model with TensorRT and ran inference on Jetson Nano/TX2. You could refer to this post:

https://devtalk.nvidia.com/default/topic/1050377/deep-learning-inference-benchmarking-instructions/?offset=66#5395624

Thanks I will try that!

Just took a quick look at it I was also wondering how I can print the mAP after training!
Its not showing in my tensorboard.

How did you printed it to terminal and tensorboard?

I created the eval_ssd.py script (https://github.com/jkjung-avt/tensorrt_demos/blob/master/eval_ssd.py) for checking mAP of the TensorRT optimized SSD COCO models.

You could read the following document and blog post for details.

https://github.com/jkjung-avt/tensorrt_demos/blob/master/README_eval_ssd.md
https://jkjung-avt.github.io/trt-detection-map/