Mask Rcnn Inference Error

I trained Mask Rcnn network. but inference task gives the following error: But I am not pruned the model

/workspace/tlt/tlt_experiments/tlt_algorithms/all_segmentation_approaches
inference with best model
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

Traceback (most recent call last):
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/inference.py", line 351, in <module>
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/inference.py", line 343, in main
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/inference.py", line 183, in infer
**AssertionError: The pruned model must be retrained first.**
Using TensorFlow backend.
write inference results to /workspace/tlt/results/tlt_mask_rcnn_corrosion1000_resnet101/infer/

Please retrained firstly.

But I did not prune the model. I just train it

Can you double check?

Yes, I checked but the result is same

I rename the model to final_model.tlt after train.
But I realized that the model name should be in this format: model.step-10.tlt
But for all TLT models, I always do this renaming.

So, you can run inference against model.step-10.tlt , but failed in running inference against final_model.tlt ?

Yes. But that does not make sense.
I need to save the model with final_model.tlt name

OK, this is a limitation for the renaming.
Currently, there is assertion for the model name.

assert ‘step’ in args.model_path, “The pruned model must be retrained first.”

Thanks for your help

Please note that “saving the model to final_step_model.tlt.” will be incorrect, because the code checks the step number of the tlt model and matches the number to the unzipped checkpoint.

Thus, please do not rename the model. Please create new directory to host the model.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.