Update:
When I convert my .pb model to an .onxx file with the command:
python3 -m tf2onnx.convert --saved-model /path_to_pbmodel --output model.onnx
I get this error message:
2021-04-07 09:39:16.584977: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPREC ATION_WARNINGS=1 to re-enable them.
/usr/lib/python3.6/runpy.py:125: RuntimeWarning: 'tf2onnx.convert' found in sys. modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.co nvert'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
WARNING:tensorflow:From /home/pi21/.local/lib/python3.6/site-packages/tf2onnx/ve rbose_logging.py:76: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.
2021-04-07 09:39:21,257 - WARNING - From /home/pi21/.local/lib/python3.6/site-pa ckages/tf2onnx/verbose_logging.py:76: The name tf.logging.set_verbosity is depre cated. Please use tf.compat.v1.logging.set_verbosity instead.
2021-04-07 09:39:21.303585: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-04-07 09:39:21.317950: I tensorflow/stream_executor/cuda/cuda_gpu_executor. cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.318090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1 666] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-04-07 09:39:21.318161: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-07 09:39:21.321952: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-04-07 09:39:21.325099: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-07 09:39:21.326041: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-07 09:39:21.330785: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-07 09:39:21.334901: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-04-07 09:39:21.335478: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-07 09:39:21.335764: I tensorflow/stream_executor/cuda/cuda_gpu_executor. cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.336019: I tensorflow/stream_executor/cuda/cuda_gpu_executor. cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.336143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1 794] Adding visible gpu devices: 0
2021-04-07 09:39:21.358093: W tensorflow/core/platform/profile_utils/cpu_utils.c c:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2021-04-07 09:39:21.358646: I tensorflow/compiler/xla/service/service.cc:168] XL A service 0x3c5fead0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-07 09:39:21.358717: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-04-07 09:39:21.433903: I tensorflow/stream_executor/cuda/cuda_gpu_executor. cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.434178: I tensorflow/compiler/xla/service/service.cc:168] XL A service 0x3baf20a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-07 09:39:21.434229: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2021-04-07 09:39:21.434609: I tensorflow/stream_executor/cuda/cuda_gpu_executor. cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.434713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1 666] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-04-07 09:39:21.434788: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-07 09:39:21.434869: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-04-07 09:39:21.434926: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-04-07 09:39:21.434977: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-04-07 09:39:21.435027: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-04-07 09:39:21.435078: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-04-07 09:39:21.435130: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-04-07 09:39:21.435288: I tensorflow/stream_executor/cuda/cuda_gpu_executor. cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.435473: I tensorflow/stream_executor/cuda/cuda_gpu_executor. cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:21.435535: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1 794] Adding visible gpu devices: 0
2021-04-07 09:39:21.435619: I tensorflow/stream_executor/platform/default/dso_lo ader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-04-07 09:39:23.952042: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-07 09:39:23.952126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0
2021-04-07 09:39:23.952188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N
2021-04-07 09:39:23.952567: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:23.952837: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero
2021-04-07 09:39:23.952985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 468 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2021-04-07 09:39:23,956 - WARNING - '--tag' not specified for saved_model. Using --tag serve
2021-04-07 09:39:23,956 - WARNING - '--signature_def' not provided. Using all signatures.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1348, in _run_fn
self._extend_graph()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1388, in _extend_graph
tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Preprocessor/TRTEngineOp_4: Could not satisfy explicit device specification '/device:CPU:0' because no supported kernel for CPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:CPU:0' assigned_device_name_='' resource_device_name_='' supported_device_types_=[GPU] possible_devices_=[]
TRTEngineOp: GPU
Colocation members, user-requested devices, and framework assigned devices, if any:
Preprocessor/TRTEngineOp_4 (TRTEngineOp) /device:CPU:0
Op: TRTEngineOp
Node attrs: output_shapes=[[?,?,?,3]], workspace_size_bytes=2995095, max_cached_engines_count=1, segment_func=Preprocessor/TRTEngineOp_4_native_segment[], segment_funcdef_name="", use_calibration=false, fixed_input_size=true, input_shapes=[[?,?,?,3]], OutT=[DT_FLOAT], precision_mode="FP32", static_engine=false, serialized_segment="", cached_engine_batches=[], InT=[DT_FLOAT], calibration_data=""
Registered kernels:
device='GPU'
[[{{node Preprocessor/TRTEngineOp_4}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/convert.py", line 488, in <module>
main()
File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/convert.py", line 213, in main
args.large_model, return_initialized_tables=True, return_tensors_to_rename=True)
File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 523, in from_saved_model
_from_saved_model_v1(sess, model_path, input_names, output_names, tag, signatures)
File "/home/pi21/.local/lib/python3.6/site-packages/tf2onnx/tf_loader.py", line 320, in _from_saved_model_v1
tf.tables_initializer().run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 2439, in run
_run_using_default_session(self, feed_dict, self.graph, session)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 5442, in _run_using_default_session
session.run(operation, feed_dict)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Preprocessor/TRTEngineOp_4: Could not satisfy explicit device specification '/device:CPU:0' because no supported kernel for CPU devices is available.
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:CPU:0' assigned_device_name_='' resource_device_name_='' supported_device_types_=[GPU] possible_devices_=[]
TRTEngineOp: GPU
Colocation members, user-requested devices, and framework assigned devices, if any:
Preprocessor/TRTEngineOp_4 (TRTEngineOp) /device:CPU:0
Op: TRTEngineOp
Node attrs: output_shapes=[[?,?,?,3]], workspace_size_bytes=2995095, max_cached_engines_count=1, segment_func=Preprocessor/TRTEngineOp_4_native_segment[], segment_funcdef_name="", use_calibration=false, fixed_input_size=true, input_shapes=[[?,?,?,3]], OutT=[DT_FLOAT], precision_mode="FP32", static_engine=false, serialized_segment="", cached_engine_batches=[], InT=[DT_FLOAT], calibration_data=""
Registered kernels:
device='GPU'
[[node Preprocessor/TRTEngineOp_4 (defined at usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
I optimized my model using TF-TRT and in dynamic mode at the time of conversion.
It seems that the reason is that the TensorEngines are not yet created and are indeed only created at the time of the inference.
What do you think about it?
Thanks
Paul Griffoul