So I have tried to convert a Tensorflow Saved Model (compiled by teachable machine with google) for TensorRT optimization. Hoping this will speed it up and work properly fix the memory issue above… however I am getting an error:
Converter Code:
from tensorflow.python.compiler.tensorrt import trt_convert as trt
import pathlib as plib
input_saved_model_dir = plib.Path('/home/tom/Python_Progs/Tensor/savedmodel/')
output_saved_model_dir = plib.Path('/home/tom/Python_Progs/Tensor/outmodel/')
input_saved_model_dir = str(input_saved_model_dir)
output_saved_model_dir = str(output_saved_model_dir)
converter = trt.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir)
converter.convert()
converter.save(output_saved_model_dir)
Output:
/usr/bin/python3 /home/tom/Python_Progs/Tensor/convSavedModel.py
2019-12-05 15:44:35.792007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:44:42.490310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.5
WARNING:tensorflow:TensorRT mismatch. Compiled against version 5.0.6, but loaded 5.1.6. Things may not work
2019-12-05 15:45:02.995995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-05 15:45:03.039523: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.039697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:03.039774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:03.039883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:03.055304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:03.093331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:03.109549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:03.124828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:03.125019: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:03.125255: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.125521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.125648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:03.128144: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.128302: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:03.128386: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:03.128449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:03.128506: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:03.128563: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:03.128616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:03.128664: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:03.128711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:03.128891: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.129111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:03.129198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:03.129300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:15.797571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-05 15:45:15.802419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-05 15:45:15.802447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-05 15:45:15.808713: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:15.808989: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:15.809209: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:15.817460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 143 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1781: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
WARNING:tensorflow:Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2019-12-05 15:45:34.273897: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.274067: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2019-12-05 15:45:34.274297: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-12-05 15:45:34.275038: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.275173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:34.275361: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:34.275456: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:34.275523: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:34.275572: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:34.275620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:34.275666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:34.275709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:34.275861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.276050: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.276126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:34.288217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-05 15:45:34.288293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-05 15:45:34.288319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-05 15:45:34.288569: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.288822: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:34.288945: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 143 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2019-12-05 15:45:34.872098: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: graph_to_optimize
2019-12-05 15:45:34.872190: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] function_optimizer: function_optimizer did nothing. time = 0.005ms.
2019-12-05 15:45:34.872223: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] function_optimizer: function_optimizer did nothing. time = 0.001ms.
2019-12-05 15:45:42.493717: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.493950: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2019-12-05 15:45:42.494158: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-12-05 15:45:42.494981: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.495138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-05 15:45:42.495362: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-05 15:45:42.495472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-05 15:45:42.495569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-05 15:45:42.495653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-05 15:45:42.495733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-05 15:45:42.495810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-05 15:45:42.495889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-05 15:45:42.496183: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.496449: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.496529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-05 15:45:42.496606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-05 15:45:42.496638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-05 15:45:42.496668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-05 15:45:42.496860: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.497257: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-05 15:45:42.497424: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 143 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2019-12-05 15:45:43.844906: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 4 ops of 3 different types in the graph that are not converted to TensorRT: Softmax, NoOp, Placeholder, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2019-12-05 15:45:43.974022: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:633] Number of TensorRT candidate segments: 1
2019-12-05 15:45:44.037919: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_0 added for segment 0 consisting of 430 nodes succeeded.
2019-12-05 15:45:44.468514: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2019-12-05 15:45:44.589282: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2019-12-05 15:45:44.589380: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 430 nodes (-263), 439 edges (-263), time = 170.223ms.
2019-12-05 15:45:44.589409: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 437 nodes (7), 446 edges (7), time = 206.055ms.
2019-12-05 15:45:44.589440: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 432 nodes (-5), 441 edges (-5), time = 136.568ms.
2019-12-05 15:45:44.589469: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 3 nodes (-429), 2 edges (-439), time = 296.231ms.
2019-12-05 15:45:44.589493: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 3 nodes (0), 2 edges (0), time = 53.271ms.
2019-12-05 15:45:44.589520: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: TRTEngineOp_0_native_segment
2019-12-05 15:45:44.589549: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 432 nodes (0), 441 edges (0), time = 48.034ms.
2019-12-05 15:45:44.589578: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 432 nodes (0), 441 edges (0), time = 60.248ms.
2019-12-05 15:45:44.589605: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 432 nodes (0), 441 edges (0), time = 53.446ms.
2019-12-05 15:45:44.589626: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 432 nodes (0), 441 edges (0), time = 6.602ms.
2019-12-05 15:45:44.589647: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 432 nodes (0), 441 edges (0), time = 57.387ms.
2019-12-05 15:46:17.035132: W tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at trt_engine_resource_ops.cc:183 : Not found: Container TF-TRT does not exist. (Could not find resource: TF-TRT/TRTEngineOp_0)
Traceback (most recent call last):
File "/home/tom/Python_Progs/Tensor/convSavedModel.py", line 12, in <module>
converter.save(output_saved_model_dir)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/compiler/tensorrt/trt_convert.py", line 1069, in save
save.save(self._saved_model, output_saved_model_dir, signatures)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 872, in save
signatures = signature_serialization.canonicalize_signatures(signatures)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/signature_serialization.py", line 119, in canonicalize_signatures
**tensor_spec_signature)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 776, in get_concrete_function
self._initialize(args, kwargs, add_initializers_to=initializer_map)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 408, in _initialize
*args, **kwds))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1848, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
return weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 905, in wrapper
raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:
relative to /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model:
signature_serialization.py:107 signature_wrapper *
return _normalize_outputs(
signature_serialization.py:156 _normalize_outputs
.format(value, key, compat.as_str_any(function_name)))
ValueError: Got a dictionary containing non-Tensor value None for key __saved_model_init_op in the output of the function __inference_pruned_9207 used to generate a SavedModel signature. Dictionaries outputs for functions used as signatures should have one Tensor output per string key.