TensorRT Version: 7.2.2-1+cuda11.1
GPU Type: dGPU (2080TI)
Nvidia Driver Version: 510.47.03
CUDA Version: 11.1
CUDNN Version:
Operating System + Version: Ubuntu 18.04.5 LTS (Inside container), System OS : Ubuntu 20.04
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 2.6.0
Baremetal or Container (if container which image + tag): Deepstream-5.1
I am trying to convert tensorflow model to tensorRT using TF-TRT but it is not generating .engine file.
Script
from tensorflow.python.compiler.tensorrt import trt_convert as trt
input_saved_model_dir = './inference_graph/saved_model/'
output_saved_model_dir = './inference_graph/trt/'
converter = trt.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir)
converter.convert()
converter.save(output_saved_model_dir)
Terminal Output :
2022-05-18 07:51:46.665718: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:46.839759: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:46.840311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:46.842332: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-18 07:51:46.843809: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:46.844309: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:46.844745: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:48.387809: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:48.390089: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:48.390194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:48.392196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9102 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
2022-05-18 07:51:48.502349: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
2022-05-18 07:51:50.610188: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:50.614551: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2022-05-18 07:51:50.614805: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2022-05-18 07:51:50.616141: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:50.616463: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:50.616699: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:50.617041: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:50.617285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:50.617487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9102 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
2022-05-18 07:51:50.901232: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1137] Optimization results for grappler item: graph_to_optimize
function_optimizer: function_optimizer did nothing. time = 0.025ms.
function_optimizer: function_optimizer did nothing. time = 0.003ms.
2022-05-18 07:51:54.153778: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:54.153928: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2022-05-18 07:51:54.154004: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2022-05-18 07:51:54.154181: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:54.154282: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:54.154384: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:54.154513: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:54.154604: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-05-18 07:51:54.154682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9102 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
2022-05-18 07:51:54.752113: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:790] There are 423 ops of 52 different types in the graph that are not converted to TensorRT: Select, Where, Split, TopKV2, Greater, Size, GatherV2, Fill, Tile, Add, ResizeBilinear, Assert, Exit, LoopCond, Equal, Exp, Merge, NextIteration, Switch, Maximum, StridedSlice, Pack, NoOp, TensorArrayV3, Placeholder, TensorArraySizeV3, TensorArrayWriteV3, Cast, Squeeze, Enter, Minimum, Shape, TensorArrayScatterV3, NonMaxSuppressionV3, Reshape, TensorArrayReadV3, Identity, ExpandDims, TensorArrayGatherV3, Pad, Slice, LogicalAnd, Mul, Sub, Const, ConcatV2, Unpack, Less, DataFormatVecPermute, Range, ZerosLike, Transpose, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2022-05-18 07:51:54.765497: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:759] Number of TensorRT candidate segments: 3
2022-05-18 07:51:54.818304: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 0 consisting of 354 nodes by TRTEngineOp_0_0.
2022-05-18 07:51:54.818915: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 1 consisting of 4 nodes by TRTEngineOp_0_1.
2022-05-18 07:51:54.818971: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 2 consisting of 5 nodes by TRTEngineOp_0_2.
2022-05-18 07:51:55.002719: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1137] Optimization results for grappler item: tf_graph
constant_folding: Graph size after: 1029 nodes (-1179), 1289 edges (-1336), time = 166.001ms.
layout: Graph size after: 1067 nodes (38), 1327 edges (38), time = 65.326ms.
constant_folding: Graph size after: 1067 nodes (0), 1327 edges (0), time = 26.188ms.
TensorRTOptimizer: Graph size after: 707 nodes (-360), 966 edges (-361), time = 151.169ms.
constant_folding: Graph size after: 706 nodes (-1), 966 edges (0), time = 16.461ms.
Optimization results for grappler item: TRTEngineOp_0_2_native_segment
constant_folding: Graph size after: 12 nodes (0), 11 edges (0), time = 0.888ms.
layout: Graph size after: 12 nodes (0), 11 edges (0), time = 0.197ms.
constant_folding: Graph size after: 12 nodes (0), 11 edges (0), time = 0.184ms.
TensorRTOptimizer: Graph size after: 12 nodes (0), 11 edges (0), time = 0.013ms.
constant_folding: Graph size after: 12 nodes (0), 11 edges (0), time = 0.213ms.
Optimization results for grappler item: TRTEngineOp_0_1_native_segment
constant_folding: Graph size after: 11 nodes (0), 10 edges (0), time = 0.191ms.
layout: Graph size after: 11 nodes (0), 10 edges (0), time = 0.165ms.
constant_folding: Graph size after: 11 nodes (0), 10 edges (0), time = 0.172ms.
TensorRTOptimizer: Graph size after: 11 nodes (0), 10 edges (0), time = 0.011ms.
constant_folding: Graph size after: 11 nodes (0), 10 edges (0), time = 0.187ms.
Optimization results for grappler item: TRTEngineOp_0_0_native_segment
constant_folding: Graph size after: 373 nodes (0), 372 edges (0), time = 18.033ms.
layout: Graph size after: 373 nodes (0), 372 edges (0), time = 34.328ms.
constant_folding: Graph size after: 373 nodes (0), 372 edges (0), time = 20.801ms.
TensorRTOptimizer: Graph size after: 373 nodes (0), 372 edges (0), time = 4.74ms.
constant_folding: Graph size after: 373 nodes (0), 372 edges (0), time = 18.486ms.
2022-05-18 07:52:27.535532: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at trt_engine_resource_ops.cc:193 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_0)
2022-05-18 07:52:27.536811: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at trt_engine_resource_ops.cc:193 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_1)
2022-05-18 07:52:27.537482: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at trt_engine_resource_ops.cc:193 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0_2)
I am attaching .pb and .pbtxt file below.
label_map.pbtxt (39 Bytes)
pipeline.config (4.1 KB)
saved_model.pb (21.6 MB)
Thanks.