mobilenet onnx problem

skywo1f · January 17, 2020, 8:12pm

I have a “mobilenet” which I froze (pb). I tried to convert it to uff, but it complains about “Unsupported operation _FusedBatchNormV3” which sounds like is not supported in uff yet. So I am now trying pb->onnx. Unfortunately I am not having any better luck with that:

~/convertToONNX$ python3 -m tf2onnx.convert  --input /home/xavier/convertToONNX/frozen_inference_graph.pb --inputs X:0 --outputs output:0 --output /home/xavier/convertToONNX/ssd-raccoonnet.onnx
2020-01-17 15:08:24.026031: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/verbose_logging.py:72: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-01-17 15:08:27,830 - WARNING - From /usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/verbose_logging.py:72: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

2020-01-17 15:08:27.835344: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-01-17 15:08:27.839512: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:27.839677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.377
pciBusID: 0000:00:00.0
2020-01-17 15:08:27.839736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-01-17 15:08:27.841502: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-01-17 15:08:27.843120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-01-17 15:08:27.843710: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-01-17 15:08:27.846533: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-01-17 15:08:27.848348: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-01-17 15:08:27.854380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-17 15:08:27.854585: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:27.854780: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:27.854956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-01-17 15:08:27.872648: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-01-17 15:08:27.873562: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2f66c8f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-17 15:08:27.873644: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-01-17 15:08:27.934881: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:27.935306: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2f7a07b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-01-17 15:08:27.935435: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Xavier, Compute Capability 7.2
2020-01-17 15:08:27.935901: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:27.936054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Xavier major: 7 minor: 2 memoryClockRate(GHz): 1.377
pciBusID: 0000:00:00.0
2020-01-17 15:08:27.936110: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-01-17 15:08:27.936152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-01-17 15:08:27.936190: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-01-17 15:08:27.936220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-01-17 15:08:27.936250: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-01-17 15:08:27.936279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-01-17 15:08:27.936311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-01-17 15:08:27.936427: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:27.936568: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:27.936639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-01-17 15:08:27.936699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-01-17 15:08:28.648073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-17 15:08:28.648189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-01-17 15:08:28.648230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-01-17 15:08:28.648539: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:28.648741: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:950] ARM64 does not support NUMA - returning NUMA node zero
2020-01-17 15:08:28.648915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8514 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/convert.py", line 161, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/convert.py", line 116, in main
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/loader.py", line 64, in from_graphdef
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/loader.py", line 37, in freeze_session
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py", line 277, in convert_variables_to_constants
    inference_graph = extract_sub_graph(input_graph_def, output_node_names)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py", line 197, in extract_sub_graph
    _assert_nodes_are_present(name_to_node, dest_nodes)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/graph_util_impl.py", line 152, in _assert_nodes_are_present
    assert d in name_to_node, "%s is not in graph" % d
AssertionError: output is not in graph

any ideas?

skywo1f · January 17, 2020, 9:16pm

Using bazel to compile summarize graph and then using summarize graph on my frozen pb, I was able to find the correct names for inputs and outputs :
image_tensor & detection_boxes, respectively. Running that, I got past that part of the converter, but now It is saying:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/tfonnx.py", line 354, in tensorflow_onnx_mapping
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/onnx_opset/tensor.py", line 321, in version_1
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/graph_builder.py", line 40, in make_slice
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/graph_builder.py", line 106, in convert_to_attribute
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/graph.py", line 260, in get_tensor_value
ValueError: get tensor value: MultipleGridAnchorGenerator/Meshgrid_9/ExpandedShape_1/ExpandDims must be Const

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/convert.py", line 161, in <module>
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/convert.py", line 145, in main
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/tfonnx.py", line 573, in process_tf_graph
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/tfonnx.py", line 357, in tensorflow_onnx_mapping
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx-1.6.0-py3.6.egg/tf2onnx/graph.py", line 175, in summary
AttributeError: 'NoneType' object has no attribute 'get_node_by_output'

skywo1f · January 17, 2020, 9:39pm

by adding --fold_const --opset 11 at the end, I was able to get it to create an .onnx file but now I am stuck. Running the detect-net with it seems to have a problem with uint8 data types:

----------------------------------------------------------------
Input filename:   /usr/local/bin/networks/SSD-Raccoonnet/ssd_raccoon.onnx
ONNX IR version:  0.0.6
Opset version:    11
Producer name:    tf2onnx
Producer version: 1.6.0
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
WARNING: ONNX model has a newer ir_version (0.0.6) than this parser was built against (0.0.3).
Unsupported ONNX data type: UINT8 (2)
ERROR: ModelImporter.cpp:54 In function importInput:
[8] Assertion failed: convert_dtype(onnx_tensor_type.elem_type(), &trt_dtype)
[TRT]   failed to parse ONNX model '/usr/local/bin/networks/SSD-Raccoonnet/ssd_raccoon.onnx'
[TRT]   device GPU, failed to load networks/SSD-Raccoonnet/ssd_raccoon.onnx
detectNet -- failed to initialize.

NVES_R · January 18, 2020, 12:10am

Hi skywolf,

TensorRT only supports FP32, FP16, INT32, and INT8: https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/infer/FoundationalTypes/DataType.html#tensorrt.DataType

UINT8 is not supported as mentioned there. Can you use int8 instead of uint8 in your model before converting your model to ONNX?

ref: https://devtalk.nvidia.com/default/topic/1067555/tensorrt/tensorrt-inference-error-while-load-onnx-model

skywo1f · January 18, 2020, 3:34pm

Is there a script that I can run on my pb file which will “parse” it and fix the uint8’s?

NVES_R · January 18, 2020, 11:00pm

I was thinking the same thing at first, but unfortunately I don’t think it’s that simple. If you have UINT8 weights with values outside (128-255) of the range of INT8 (-128-127), It will likely alter the results of your network. So I believe the network has to be trained with the types set to INT8 in order to learn the correct weights and be compatible with TensorRT at the moment.

Turowicz · June 5, 2020, 11:26am

@NVES_R In other words, TensorRT can’t run any of the TF Object Detection models from Zoo?

jmak · June 5, 2020, 10:35pm

int32 would work, uint8 isn’t supported in TensorRT and int8 can’t represent the range of values that uint8 can, but int32 or float values would represent all the pixel values correctly and let you use the same weights.

sdy1029 · July 20, 2020, 12:30am

hello. I used tf.NodeDef to solve same uint8 issue in ‘mobilenet_v2_coco(pb) → onnx → trt’ process. Input node(image_tensor’s dtype is uint8) issue seems to solve, but it has other problem. Network model(pb) have two nodes dtype is int64. that node is tf.where (Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ClipToWindow/Where:0). anyone have solution?

I change uint8 node to float32 node by tf.NodeDef…uint8(0 - 255) to float32(0.0 -255.0). Original next node is tf.cast to float32 from uint8, now i change float32 to float32. it means pass-through…is this right?

Turowicz · November 19, 2020, 1:39pm

This should help: Unsupported ONNX data type: UINT8 (2) · Issue #400 · onnx/onnx-tensorrt · GitHub

Topic		Replies	Views
Cannot convert SSD ONNX model to TensorRT TensorRT tensorrt	15	2351	November 23, 2022
Problem converting ONNX model to TensorRT Engine for SSD Mobilenet V2 Jetson Nano tensorrt , nvbugs , ssd , onnx	38	8745	October 18, 2021
Unsupported ONNX data type: UINT8 (2) TensorRT	24	8907	May 6, 2021
Running a pytorch network converted to ONNX with TensorRT on the TX2 Jetson TX2	24	8852	October 18, 2021
I am trying to convert the ONNX SSD mobilnet v3 model into TensorRT Engine. I am getting the below error Jetson TX2 tensorrt , tensorflow	24	3687	February 17, 2022
Use pre-trained object detection TF2 models with TensorRT ONNX TensorRT	9	1912	May 31, 2021
Tensorflow MobilenetSSD v2 conversion in TensorRT TensorRT tensorrt , tensorflow	3	998	March 8, 2021
Unable to load parse onnx network with int8 operations TensorRT tensorrt , onnx	7	2029	April 14, 2021
Error code 4 internal error unnamed layer TensorRT cudnn	9	1346	October 4, 2024
I do not get any performance improvement after using TensorRT provider for object detection model Jetson Nano tensorrt , onnx	7	1396	July 12, 2022

mobilenet onnx problem

Related topics