Description
I am trying to perform inference of an SSD_MobileNet_V2 frozen graph inside a docker container (tensorflow:19.12-tf1-py3) . Here is the code that I have used to run load the model.
with tf.Session() as sess:
# First deserialize your frozen graph:
with tf.gfile.GFile(path, 'rb') as f:
frozen_graph = tf.GraphDef()
frozen_graph.ParseFromString(f.read())
# Now you can create a TensorRT inference graph from your
# frozen graph:
converter = trt.TrtGraphConverter(
input_graph_def=frozen_graph,
nodes_blacklist=output_node_names, #output nodes
is_dynamic_op=True)
trt_graph = converter.convert()
# saved_model_dir_trt = "/tensorrt-exp/frozen_graph_sets/tensorrt_model.trt"
# converter.save(saved_model_dir_trt)
# Import the TensorRT graph into a new graph and run:
output_node = tf.import_graph_def(
trt_graph,
return_elements=output_node_names)
sess.run(output_node)
When I run it, I am getting warning messages and errors which are listed below
2021-01-07 09:12:15.870404: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 440.100.0
2021-01-07 09:12:15.870433: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 440.100.0
2021-01-07 09:12:15.870444: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 440.100.0
2021-01-07 09:12:15.895578: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz
2021-01-07 09:12:15.897850: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4dd0210 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-07 09:12:15.897908: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-01-07 09:12:15.913938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2021-01-07 09:12:16.136353: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2021-01-07 09:12:16.137217: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2021-01-07 09:12:16.358155: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 402 ops of 53 different types in the graph that are not converted to TensorRT: Sum, Where, Split, TopKV2, Assert, Equal, Select, Greater, Size, GatherV2, Fill, GreaterEqual, TensorArrayWriteV3, AddV2, Exit, NoOp, Pack, LoopCond, Merge, NextIteration, Switch, TensorArraySizeV3, TensorArrayV3, Placeholder, Maximum, StridedSlice, Cast, Minimum, Shape, TensorArrayGatherV3, NonMaxSuppressionV5, TensorArrayScatterV3, Reshape, TensorArrayReadV3, FusedBatchNormV3, Enter, Squeeze, ConcatV2, Unpack, Less, Range, ZerosLike, Transpose, RealDiv, Tile, Pad, LogicalAnd, Slice, Mul, Const, Sub, ExpandDims, Identity, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2021-01-07 09:12:16.363051: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:633] Number of TensorRT candidate segments: 40
2021-01-07 09:12:16.493432: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:696] Couldn't get current device: unknown error
2021-01-07 09:12:16.493471: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493476: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0
2021-01-07 09:12:16.493539: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_0 added for segment 0 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493586: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493591: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0
2021-01-07 09:12:16.493627: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_1 added for segment 1 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493683: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493690: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0
2021-01-07 09:12:16.493745: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_2 added for segment 2 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493808: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493818: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0
2021-01-07 09:12:16.493872: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_3 added for segment 3 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.493931: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.493938: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0
2021-01-07 09:12:16.493985: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_4 added for segment 4 consisting of 11 nodes succeeded.
2021-01-07 09:12:16.494030: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.494035: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0
2021-01-07 09:12:16.494078: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_5 added for segment 5 consisting of 9 nodes succeeded.
2021-01-07 09:12:16.494115: E tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:101] Could not find any TF GPUs
2021-01-07 09:12:16.494121: W tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:723] Can't identify the cuda device. Running on device 0
Environment
TensorRT Version: TensorRT version: (6, 0, 1)
GPU Type: X86 GTX 1660 Ti
Nvidia Driver Version: 440.100
CUDA Version: 10.2
CUDNN Version:
Operating System + Version: Ubuntu + 20.04
Python Version (if applicable): 3.6
TensorFlow Version (if applicable): 1.15
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/tensorflow:19.12-tf1-py3
Relevant Files
Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
Steps To Reproduce
Please include:
- Exact steps/commands to build your repro
- Exact steps/commands to run your repro
- Full traceback of errors encountered