Hello,
I’m trying to export a MaskRCNN model using INT8 precision. My export command is the following:
tlt-export mask_rcnn -m /workspace/tlt-experiments/models/experiments/Tests/model.step-50.tlt -o /workspace/tlt-experiments/models/experiments/Tests/model_n.etlt -e /workspace/tlt-experiments/oddl_ds/models/templates/train/mask_rcnn.txt -k YnJibjZhdm9nbmJ2MTRrYnRncGM1cTZudGg6NTc2Y2Q3NDUtZTVhOC00MTBhLTg1ZjMtM2NjZDUwZDM4NzY5 --cal_image_dir /workspace/tlt-experiments/data/processed/labs_v2.0.0/images --cal_cache_file /workspace/tlt-experiments/models/experiments/Tests/cal.bin --cal_data_file /workspace/tlt-experiments/models/experiments/Tests/cal.tensor --data_type int8
However, I’m getting the following error.
Using TensorFlow backend.
2020-10-30 21:55:12,264 [INFO] /usr/local/lib/python3.6/dist-packages/iva/mask_rcnn/utils/spec_loader.pyc: Loading specification from /workspace/tlt-experiments/oddl_ds/models/templates/train/mask_rcnn.txt
2020-10-30 21:55:13,536 [INFO] root: Loading weights from /workspace/tlt-experiments/models/experiments/Tests/model.step-50.tlt
[MaskRCNN] INFO : [eval] AMP is activated - Experiment Feature
[MaskRCNN] INFO : [*] Limiting the amount of sample to: 50
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS... Scope: multilevel_propose_rois/level_2/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS... Scope: multilevel_propose_rois/level_3/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS... Scope: multilevel_propose_rois/level_4/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS... Scope: multilevel_propose_rois/level_5/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS... Scope: multilevel_propose_rois/level_6/
4 ops no flops stats due to incomplete shapes.
Parsing Inputs...
[MaskRCNN] INFO : [Inference Compute Statistics] 271.8 GFLOPS/image
WARNING: To create TensorRT plugin nodes, please use the `create_plugin_node` function instead.
WARNING: To create TensorRT plugin nodes, please use the `create_plugin_node` function instead.
NOTE: UFF has been tested with TensorFlow 1.14.0.
WARNING: The version of TensorFlow installed on this system is not guaranteed to work with UFF.
Warning: No conversion function registered for layer: MultilevelCropAndResize_TRT yet.
Converting pyramid_crop_and_resize_mask as custom op: MultilevelCropAndResize_TRT
Warning: No conversion function registered for layer: ResizeNearest_TRT yet.
Converting nearest_upsampling as custom op: ResizeNearest_TRT
Warning: No conversion function registered for layer: ResizeNearest_TRT yet.
Converting nearest_upsampling_1 as custom op: ResizeNearest_TRT
Warning: No conversion function registered for layer: ResizeNearest_TRT yet.
Converting nearest_upsampling_2 as custom op: ResizeNearest_TRT
Warning: No conversion function registered for layer: SpecialSlice_TRT yet.
Converting mrcnn_detection_bboxes as custom op: SpecialSlice_TRT
Warning: No conversion function registered for layer: GenerateDetection_TRT yet.
Converting generate_detections as custom op: GenerateDetection_TRT
Warning: No conversion function registered for layer: MultilevelProposeROI_TRT yet.
Converting multilevel_propose_rois as custom op: MultilevelProposeROI_TRT
Warning: No conversion function registered for layer: MultilevelCropAndResize_TRT yet.
Converting pyramid_crop_and_resize_box as custom op: MultilevelCropAndResize_TRT
DEBUG [/usr/local/lib/python3.6/dist-packages/uff/converters/tensorflow/converter.py:96] Marking ['generate_detections', 'mask_head/mask_fcn_logits/BiasAdd'] as outputs
2020-10-30 21:55:27,983 [INFO] iva.mask_rcnn.export.exporter: Converted model was saved into /workspace/tlt-experiments/models/experiments/Tests/model_n.etlt
2020-10-30 21:55:27,990 [INFO] iva.mask_rcnn.export.exporter: Calibration takes time especially if number of batches is large.
[TensorRT] ERROR: UffParser: Parser error: multilevel_propose_rois/level_2/Reshape_1: Reshape: Volume mismatch. Note: Initial Volume = 391680, New Volume = 414720
2020-10-30 21:55:28,546 [ERROR] modulus.export._tensorrt: Failed to parse UFF File
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/export/_tensorrt.py", line 687, in _load_from_files
Traceback (most recent call last):
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/export/_tensorrt.py", line 687, in _load_from_files
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/tlt-export", line 8, in <module>
sys.exit(main())
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/app.py", line 185, in main
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/app.py", line 263, in run_export
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/export/exporter.py", line 547, in export
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/export/_tensorrt.py", line 676, in __init__
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/export/_tensorrt.py", line 481, in __init__
File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/export/_tensorrt.py", line 695, in _load_from_files
AssertionError: UFF parsing failed on line 687 in statement
I would really appreciate some help on understanding why am I getting a volume mismatch, given that there’s been no modification from the model between training and exporting.
Thanks!