Train mask-rcnn failure

gao-yan · November 1, 2021, 1:00pm

• Hardware V100
• Network Type Mask_rcnn
• TLT Version TAO toolkit_version: 3.21.08
• Training spec file(If have, please share here)
• How to reproduce the issue ?

tao mask_rcnn train \
-e /workspace/tao-experiments/mask_rcnn/specs/maskrcnn_train_resnet50.txt \
-d /workspace/tao-experiments/mask_rcnn/exp_unpruned \
-k nvidia_tlt \
--gpus 1 \
--gpu_index 6

logs as following:

2021-11-01 20:40:27,309 [INFO] root: Registry: [‘nvcr.io’]
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/init.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/init.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

[MaskRCNN] INFO : Loading pretrained model…
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/executer/distributed_executer.py:220: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/executer/distributed_executer.py:223: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/executer/distributed_executer.py:224: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[MaskRCNN] INFO : Create EncryptCheckpointSaverHook.

[MaskRCNN] INFO : =================================
[MaskRCNN] INFO : Start training cycle 01
[MaskRCNN] INFO : =================================

WARNING:tensorflow:Entity <function InputReader.call.._prefetch_dataset at 0x7f6c8d197e18> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <function InputReader.call.._prefetch_dataset at 0x7f6c8d197e18>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:Entity <function dataset_parser at 0x7f6cb0bead90> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <function dataset_parser at 0x7f6cb0bead90>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
[MaskRCNN] INFO : ***********************
[MaskRCNN] INFO : Building model graph…
[MaskRCNN] INFO : ***********************
WARNING:tensorflow:Entity <bound method AnchorLayer.call of <iva.mask_rcnn.layers.anchor_layer.AnchorLayer object at 0x7f6c8c873f60>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method AnchorLayer.call of <iva.mask_rcnn.layers.anchor_layer.AnchorLayer object at 0x7f6c8c873f60>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method MultilevelProposal.call of <iva.mask_rcnn.layers.multilevel_proposal_layer.MultilevelProposal object at 0x7f6c87b45eb8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method MultilevelProposal.call of <iva.mask_rcnn.layers.multilevel_proposal_layer.MultilevelProposal object at 0x7f6c87b45eb8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS… Scope: MLP/multilevel_propose_rois/level_2/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS… Scope: MLP/multilevel_propose_rois/level_3/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS… Scope: MLP/multilevel_propose_rois/level_4/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS… Scope: MLP/multilevel_propose_rois/level_5/
[MaskRCNN] INFO : [ROI OPs] Using Batched NMS… Scope: MLP/multilevel_propose_rois/level_6/
WARNING:tensorflow:Entity <bound method ProposalAssignment.call of <iva.mask_rcnn.layers.proposal_assignment_layer.ProposalAssignment object at 0x7f6c87b484a8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method ProposalAssignment.call of <iva.mask_rcnn.layers.proposal_assignment_layer.ProposalAssignment object at 0x7f6c87b484a8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method MultilevelCropResize.call of <iva.mask_rcnn.layers.multilevel_crop_resize_layer.MultilevelCropResize object at 0x7f6c879ecdd8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method MultilevelCropResize.call of <iva.mask_rcnn.layers.multilevel_crop_resize_layer.MultilevelCropResize object at 0x7f6c879ecdd8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c87586438>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c87586438>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c87a208d0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c87a208d0>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c87586518>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c87586518>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method BoxTargetEncoder.call of <iva.mask_rcnn.layers.box_target_encoder.BoxTargetEncoder object at 0x7f6c875bc748>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method BoxTargetEncoder.call of <iva.mask_rcnn.layers.box_target_encoder.BoxTargetEncoder object at 0x7f6c875bc748>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method ForegroundSelectorForMask.call of <iva.mask_rcnn.layers.foreground_selector_for_mask.ForegroundSelectorForMask object at 0x7f6c87586278>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method ForegroundSelectorForMask.call of <iva.mask_rcnn.layers.foreground_selector_for_mask.ForegroundSelectorForMask object at 0x7f6c87586278>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method MultilevelCropResize.call of <iva.mask_rcnn.layers.multilevel_crop_resize_layer.MultilevelCropResize object at 0x7f6c875476d8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method MultilevelCropResize.call of <iva.mask_rcnn.layers.multilevel_crop_resize_layer.MultilevelCropResize object at 0x7f6c875476d8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c873f6048>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method ReshapeLayer.call of <iva.mask_rcnn.layers.reshape_layer.ReshapeLayer object at 0x7f6c873f6048>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:Entity <bound method MaskPostprocess.call of <iva.mask_rcnn.layers.mask_postprocess_layer.MaskPostprocess object at 0x7f6c873f6128>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method MaskPostprocess.call of <iva.mask_rcnn.layers.mask_postprocess_layer.MaskPostprocess object at 0x7f6c873f6128>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
4 ops no flops stats due to incomplete shapes.
Parsing Inputs…
[MaskRCNN] INFO : [Training Compute Statistics] 542.3 GFLOPS/image
WARNING:tensorflow:Entity <bound method MaskTargetsLayer.call of <iva.mask_rcnn.layers.mask_targets_layer.MaskTargetsLayer object at 0x7f6c867b30f0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: Unable to locate the source code of <bound method MaskTargetsLayer.call of <iva.mask_rcnn.layers.mask_targets_layer.MaskTargetsLayer object at 0x7f6c867b30f0>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md

GitHub - tensorflow/addons: Useful extra functionality for TensorFlow 2.x maintained by SIG-addons

GitHub - tensorflow/io: Dataset, streaming, and file system extensions maintained by TensorFlow SIG-IO (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

[MaskRCNN] WARNING : Checkpoint is missing variable [l2/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [l2/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [l3/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [l3/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [l4/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [l4/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [l5/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [l5/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d2/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d2/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d3/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d3/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d4/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d4/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d5/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [post_hoc_d5/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [rpn/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [rpn/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [rpn-class/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [rpn-class/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [rpn-box/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [rpn-box/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [fc6/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [fc6/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [fc7/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [fc7/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [class-predict/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [class-predict/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [box-predict/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [box-predict/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l0/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l0/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l1/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l1/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l2/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l2/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l3/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask-conv-l3/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [conv5-mask/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [conv5-mask/bias]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask_fcn_logits/kernel]
[MaskRCNN] WARNING : Checkpoint is missing variable [mask_fcn_logits/bias]
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
[MaskRCNN] INFO : ============================ GIT REPOSITORY ============================
[MaskRCNN] INFO : BRANCH NAME:
[MaskRCNN] INFO : %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[MaskRCNN] INFO : ============================ MODEL STATISTICS ===========================
[MaskRCNN] INFO : # Model Weights: 28,580,339
[MaskRCNN] INFO : # Trainable Weights: 43,997,043
[MaskRCNN] INFO : %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

[MaskRCNN] INFO : ============================ TRAINABLE VARIABLES ============= [MaskRCNN] INFO : [#0001] conv1/kernel:0 [MaskRCNN] INFO : [#0002] bn_conv1/gamma:0 [MaskRCNN] INFO : [#0003] bn_conv1/beta:0 [MaskRCNN] INFO : [#0004] block_1a_conv_1/kernel:0 [MaskRCNN] INFO : [#0005] block_1a_bn_1/gamma:0 [MaskRCNN] INFO : [#0006] block_1a_bn_1/beta:0 [MaskRCNN] INFO : [#0007] block_1a_conv_2/kernel:0 [MaskRCNN] INFO : [#0008] block_1a_bn_2/gamma:0 [MaskRCNN] INFO : [#0009] block_1a_bn_2/beta:0 [MaskRCNN] INFO : [#0010] block_1a_conv_3/kernel:0 [MaskRCNN] INFO : [#0011] block_1a_bn_3/gamma:0 [MaskRCNN] INFO : [#0012] block_1a_bn_3/beta:0 [MaskRCNN] INFO : [#0013] block_1a_conv_shortcut/kernel:0 [MaskRCNN] INFO : [#0014] block_1a_bn_shortcut/gamma:0 [MaskRCNN] INFO : [#0015] block_1a_bn_shortcut/beta:0 [MaskRCNN] INFO : [#0016] block_1b_conv_1/kernel:0 [MaskRCNN] INFO : [#0017] block_1b_bn_1/gamma:0 [MaskRCNN] INFO : [#0018] block_1b_bn_1/beta:0 [MaskRCNN] INFO : [#0019] block_1b_conv_2/kernel:0 [MaskRCNN] INFO : [#0020] block_1b_bn_2/gamma:0 [MaskRCNN] INFO : [#0021] block_1b_bn_2/beta:0 [MaskRCNN] INFO : [#0022] block_1b_conv_3/kernel:0 [MaskRCNN] INFO : [#0023] block_1b_bn_3/gamma:0 [MaskRCNN] INFO : [#0024] block_1b_bn_3/beta:0 [MaskRCNN] INFO : [#0025] block_1c_conv_1/kernel:0 [MaskRCNN] INFO : [#0026] block_1c_bn_1/gamma:0 [MaskRCNN] INFO : [#0027] block_1c_bn_1/beta:0 [MaskRCNN] INFO : [#0028] block_1c_conv_2/kernel:0 [MaskRCNN] INFO : [#0029] block_1c_bn_2/gamma:0 [MaskRCNN] INFO : [#0030] block_1c_bn_2/beta:0 [MaskRCNN] INFO : [#0031] block_1c_conv_3/kernel:0 [MaskRCNN] INFO : [#0032] block_1c_bn_3/gamma:0 [MaskRCNN] INFO : [#0033] block_1c_bn_3/beta:0 [MaskRCNN] INFO : [#0034] block_2a_conv_1/kernel:0 [MaskRCNN] INFO : [#0035] block_2a_bn_1/gamma:0 [MaskRCNN] INFO : [#0036] block_2a_bn_1/beta:0 [MaskRCNN] INFO : [#0037] block_2a_conv_2/kernel:0 [MaskRCNN] INFO : [#0038] block_2a_bn_2/gamma:0 [MaskRCNN] INFO : [#0039] block_2a_bn_2/beta:0 [MaskRCNN] INFO : [#0040] block_2a_conv_3/kernel:0 [MaskRCNN] INFO : [#0041] block_2a_bn_3/gamma:0 [MaskRCNN] INFO : [#0042] block_2a_bn_3/beta:0 [MaskRCNN] INFO : [#0043] block_2a_conv_shortcut/kernel:0 [MaskRCNN] INFO : [#0044] block_2a_bn_shortcut/gamma:0 [MaskRCNN] INFO : [#0045] block_2a_bn_shortcut/beta:0 [MaskRCNN] INFO : [#0046] block_2b_conv_1/kernel:0 [MaskRCNN] INFO : [#0047] block_2b_bn_1/gamma:0 [MaskRCNN] INFO : [#0048] block_2b_bn_1/beta:0 [MaskRCNN] INFO : [#0049] block_2b_conv_2/kernel:0 [MaskRCNN] INFO : [#0050] block_2b_bn_2/gamma:0 [MaskRCNN] INFO : [#0051] block_2b_bn_2/beta:0 [MaskRCNN] INFO : [#0052] block_2b_conv_3/kernel:0 [MaskRCNN] INFO : [#0053] block_2b_bn_3/gamma:0 [MaskRCNN] INFO : [#0054] block_2b_bn_3/beta:0 [MaskRCNN] INFO : [#0055] block_2c_conv_1/kernel:0 [MaskRCNN] INFO : [#0056] block_2c_bn_1/gamma:0 [MaskRCNN] INFO : [#0057] block_2c_bn_1/beta:0 [MaskRCNN] INFO : [#0058] block_2c_conv_2/kernel:0 [MaskRCNN] INFO : [#0059] block_2c_bn_2/gamma:0 [MaskRCNN] INFO : [#0060] block_2c_bn_2/beta:0 [MaskRCNN] INFO : [#0061] block_2c_conv_3/kernel:0 [MaskRCNN] INFO : [#0062] block_2c_bn_3/gamma:0 [MaskRCNN] INFO : [#0063] block_2c_bn_3/beta:0 [MaskRCNN] INFO : [#0064] block_2d_conv_1/kernel:0 [MaskRCNN] INFO : [#0065] block_2d_bn_1/gamma:0 [MaskRCNN] INFO : [#0066] block_2d_bn_1/beta:0 [MaskRCNN] INFO : [#0067] block_2d_conv_2/kernel:0 [MaskRCNN] INFO : [#0068] block_2d_bn_2/gamma:0 [MaskRCNN] INFO : [#0069] block_2d_bn_2/beta:0 [MaskRCNN] INFO : [#0070] block_2d_conv_3/kernel:0 [MaskRCNN] INFO : [#0071] block_2d_bn_3/gamma:0 [MaskRCNN] INFO : [#0072] block_2d_bn_3/beta:0 [MaskRCNN] INFO : [#0073] block_3a_conv_1/kernel:0 [MaskRCNN] INFO : [#0074] block_3a_bn_1/gamma:0 [MaskRCNN] INFO : [#0075] block_3a_bn_1/beta:0 [MaskRCNN] INFO : [#0076] block_3a_conv_2/kernel:0 [MaskRCNN] INFO : [#0077] block_3a_bn_2/gamma:0 [MaskRCNN] INFO : [#0078] block_3a_bn_2/beta:0 [MaskRCNN] INFO : [#0079] block_3a_conv_3/kernel:0 [MaskRCNN] INFO : [#0080] block_3a_bn_3/gamma:0 [MaskRCNN] INFO : [#0081] block_3a_bn_3/beta:0 [MaskRCNN] INFO : [#0082] block_3a_conv_shortcut/kernel:0 [MaskRCNN] INFO : [#0083] block_3a_bn_shortcut/gamma:0 [MaskRCNN] INFO : [#0084] block_3a_bn_shortcut/beta:0 [MaskRCNN] INFO : [#0085] block_3b_conv_1/kernel:0 [MaskRCNN] INFO : [#0086] block_3b_bn_1/gamma:0 [MaskRCNN] INFO : [#0087] block_3b_bn_1/beta:0 [MaskRCNN] INFO : [#0088] block_3b_conv_2/kernel:0 [MaskRCNN] INFO : [#0089] block_3b_bn_2/gamma:0 [MaskRCNN] INFO : [#0090] block_3b_bn_2/beta:0 [MaskRCNN] INFO : [#0091] block_3b_conv_3/kernel:0 [MaskRCNN] INFO : [#0092] block_3b_bn_3/gamma:0 [MaskRCNN] INFO : [#0093] block_3b_bn_3/beta:0 [MaskRCNN] INFO : [#0094] block_3c_conv_1/kernel:0 [MaskRCNN] INFO : [#0095] block_3c_bn_1/gamma:0 [MaskRCNN] INFO : [#0096] block_3c_bn_1/beta:0 [MaskRCNN] INFO : [#0097] block_3c_conv_2/kernel:0 [MaskRCNN] INFO : [#0098] block_3c_bn_2/gamma:0 [MaskRCNN] INFO : [#0099] block_3c_bn_2/beta:0 [MaskRCNN] INFO : [#0100] block_3c_conv_3/kernel:0 [MaskRCNN] INFO : [#0101] block_3c_bn_3/gamma:0 [MaskRCNN] INFO : [#0102] block_3c_bn_3/beta:0 [MaskRCNN] INFO : [#0103] block_3d_conv_1/kernel:0 [MaskRCNN] INFO : [#0104] block_3d_bn_1/gamma:0 [MaskRCNN] INFO : [#0105] block_3d_bn_1/beta:0 [MaskRCNN] INFO : [#0106] block_3d_conv_2/kernel:0 [MaskRCNN] INFO : [#0107] block_3d_bn_2/gamma:0 [MaskRCNN] INFO : [#0108] block_3d_bn_2/beta:0 [MaskRCNN] INFO : [#0109] block_3d_conv_3/kernel:0 [MaskRCNN] INFO : [#0110] block_3d_bn_3/gamma:0 [MaskRCNN] INFO : [#0111] block_3d_bn_3/beta:0 [MaskRCNN] INFO : [#0112] block_3e_conv_1/kernel:0 [MaskRCNN] INFO : [#0113] block_3e_bn_1/gamma:0 [MaskRCNN] INFO : [#0114] block_3e_bn_1/beta:0 [MaskRCNN] INFO : [#0115] block_3e_conv_2/kernel:0 [MaskRCNN] INFO : [#0116] block_3e_bn_2/gamma:0 [MaskRCNN] INFO : [#0117] block_3e_bn_2/beta:0 [MaskRCNN] INFO : [#0118] block_3e_conv_3/kernel:0 [MaskRCNN] INFO : [#0119] block_3e_bn_3/gamma:0 [MaskRCNN] INFO : [#0120] block_3e_bn_3/beta:0 [MaskRCNN] INFO : [#0121] block_3f_conv_1/kernel:0 [MaskRCNN] INFO : [#0122] block_3f_bn_1/gamma:0 [MaskRCNN] INFO : [#0123] block_3f_bn_1/beta:0 [MaskRCNN] INFO : [#0124] block_3f_conv_2/kernel:0 [MaskRCNN] INFO : [#0125] block_3f_bn_2/gamma:0 [MaskRCNN] INFO : [#0126] block_3f_bn_2/beta:0 [MaskRCNN] INFO : [#0127] block_3f_conv_3/kernel:0 [MaskRCNN] INFO : [#0128] block_3f_bn_3/gamma:0 [MaskRCNN] INFO : [#0129] block_3f_bn_3/beta:0 [MaskRCNN] INFO : [#0130] block_4a_conv_1/kernel:0 [MaskRCNN] INFO : [#0131] block_4a_bn_1/gamma:0 [MaskRCNN] INFO : [#0132] block_4a_bn_1/beta:0 [MaskRCNN] INFO : [#0133] block_4a_conv_2/kernel:0 [MaskRCNN] INFO : [#0134] block_4a_bn_2/gamma:0 [MaskRCNN] INFO : [#0135] block_4a_bn_2/beta:0 [MaskRCNN] INFO : [#0136] block_4a_conv_3/kernel:0 [MaskRCNN] INFO : [#0137] block_4a_bn_3/gamma:0 [MaskRCNN] INFO : [#0138] block_4a_bn_3/beta:0 [MaskRCNN] INFO : [#0139] block_4a_conv_shortcut/kernel:0 [MaskRCNN] INFO : [#0140] block_4a_bn_shortcut/gamma:0 [MaskRCNN] INFO : [#0141] block_4a_bn_shortcut/beta:0 [MaskRCNN] INFO : [#0142] block_4b_conv_1/kernel:0 [MaskRCNN] INFO : [#0143] block_4b_bn_1/gamma:0 [MaskRCNN] INFO : [#0144] block_4b_bn_1/beta:0 [MaskRCNN] INFO : [#0145] block_4b_conv_2/kernel:0 [MaskRCNN] INFO : [#0146] block_4b_bn_2/gamma:0 [MaskRCNN] INFO : [#0147] block_4b_bn_2/beta:0 [MaskRCNN] INFO : [#0148] block_4b_conv_3/kernel:0 [MaskRCNN] INFO : [#0149] block_4b_bn_3/gamma:0 [MaskRCNN] INFO : [#0150] block_4b_bn_3/beta:0 [MaskRCNN] INFO : [#0151] block_4c_conv_1/kernel:0 [MaskRCNN] INFO : [#0152] block_4c_bn_1/gamma:0 [MaskRCNN] INFO : [#0153] block_4c_bn_1/beta:0 [MaskRCNN] INFO : [#0154] block_4c_conv_2/kernel:0 [MaskRCNN] INFO : [#0155] block_4c_bn_2/gamma:0 [MaskRCNN] INFO : [#0156] block_4c_bn_2/beta:0 [MaskRCNN] INFO : [#0157] block_4c_conv_3/kernel:0 [MaskRCNN] INFO : [#0158] block_4c_bn_3/gamma:0 [MaskRCNN] INFO : [#0159] block_4c_bn_3/beta:0 [MaskRCNN] INFO : [#0160] l2/kernel:0 [MaskRCNN] INFO : [#0161] l2/bias:0 [MaskRCNN] INFO : [#0162] l3/kernel:0 [MaskRCNN] INFO : [#0163] l3/bias:0 [MaskRCNN] INFO : [#0164] l4/kernel:0 [MaskRCNN] INFO : [#0165] l4/bias:0 [MaskRCNN] INFO : [#0166] l5/kernel:0 [MaskRCNN] INFO : [#0167] l5/bias:0 [MaskRCNN] INFO : [#0168] post_hoc_d2/kernel:0 [MaskRCNN] INFO : [#0169] post_hoc_d2/bias:0 [MaskRCNN] INFO : [#0170] post_hoc_d3/kernel:0 [MaskRCNN] INFO : [#0171] post_hoc_d3/bias:0 [MaskRCNN] INFO : [#0172] post_hoc_d4/kernel:0 [MaskRCNN] INFO : [#0173] post_hoc_d4/bias:0 [MaskRCNN] INFO : [#0174] post_hoc_d5/kernel:0 [MaskRCNN] INFO : [#0175] post_hoc_d5/bias:0 [MaskRCNN] INFO : [#0176] rpn/kernel:0 [MaskRCNN] INFO : [#0177] rpn/bias:0 [MaskRCNN] INFO : [#0178] rpn-class/kernel:0 [MaskRCNN] INFO : [#0179] rpn-class/bias:0 [MaskRCNN] INFO : [#0180] rpn-box/kernel:0 [MaskRCNN] INFO : [#0181] rpn-box/bias:0 [MaskRCNN] INFO : [#0182] fc6/kernel:0 [MaskRCNN] INFO : [#0183] fc6/bias:0 [MaskRCNN] INFO : [#0184] fc7/kernel:0 [MaskRCNN] INFO : [#0185] fc7/bias:0 [MaskRCNN] INFO : [#0186] class-predict/kernel:0 [MaskRCNN] INFO : [#0187] class-predict/bias:0 [MaskRCNN] INFO : [#0188] box-predict/kernel:0 [MaskRCNN] INFO : [#0189] box-predict/bias:0 [MaskRCNN] INFO : [#0190] mask-conv-l0/kernel:0 [MaskRCNN] INFO : [#0191] mask-conv-l0/bias:0 [MaskRCNN] INFO : [#0192] mask-conv-l1/kernel:0 [MaskRCNN] INFO : [#0193] mask-conv-l1/bias:0 [MaskRCNN] INFO : [#0194] mask-conv-l2/kernel:0 [MaskRCNN] INFO : [#0195] mask-conv-l2/bias:0 [MaskRCNN] INFO : [#0196] mask-conv-l3/kernel:0 [MaskRCNN] INFO : [#0197] mask-conv-l3/bias:0 [MaskRCNN] INFO : [#0198] conv5-mask/kernel:0 [MaskRCNN] INFO : [#0199] conv5-mask/bias:0 [MaskRCNN] INFO : [#0200] mask_fcn_logits/kernel:0 [MaskRCNN] INFO : [#0201] mask_fcn_logits/bias:0 [MaskRCNN] INFO : %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% ===========
=> (7, 7, 3, 64)
=> (64,)
=> (64,)
=> (1, 1, 64, 64)
=> (64,)
=> (64,)
=> (3, 3, 64, 64)
=> (64,)
=> (64,)
=> (1, 1, 64, 256)
=> (256,)
=> (256,)
=> (1, 1, 64, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 64)
=> (64,)
=> (64,)
=> (3, 3, 64, 64)
=> (64,)
=> (64,)
=> (1, 1, 64, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 64)
=> (64,)
=> (64,)
=> (3, 3, 64, 64)
=> (64,)
=> (64,)
=> (1, 1, 64, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 128)
=> (128,)
=> (128,)
=> (3, 3, 128, 128)
=> (128,)
=> (128,)
=> (1, 1, 128, 512)
=> (512,)
=> (512,)
=> (1, 1, 256, 512)
=> (512,)
=> (512,)
=> (1, 1, 512, 128)
=> (128,)
=> (128,)
=> (3, 3, 128, 128)
=> (128,)
=> (128,)
=> (1, 1, 128, 512)
=> (512,)
=> (512,)
=> (1, 1, 512, 128)
=> (128,)
=> (128,)
=> (3, 3, 128, 128)
=> (128,)
=> (128,)
=> (1, 1, 128, 512)
=> (512,)
=> (512,)
=> (1, 1, 512, 128)
=> (128,)
=> (128,)
=> (3, 3, 128, 128)
=> (128,)
=> (128,)
=> (1, 1, 128, 512)
=> (512,)
=> (512,)
=> (1, 1, 512, 256)
=> (256,)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 1024)
=> (1024,)
=> (1024,)
=> (1, 1, 512, 1024)
=> (1024,)
=> (1024,)
=> (1, 1, 1024, 256)
=> (256,)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 1024)
=> (1024,)
=> (1024,)
=> (1, 1, 1024, 256)
=> (256,)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 1024)
=> (1024,)
=> (1024,)
=> (1, 1, 1024, 256)
=> (256,)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 1024)
=> (1024,)
=> (1024,)
=> (1, 1, 1024, 256)
=> (256,)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 1024)
=> (1024,)
=> (1024,)
=> (1, 1, 1024, 256)
=> (256,)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (256,)
=> (1, 1, 256, 1024)
=> (1024,)
=> (1024,)
=> (1, 1, 1024, 512)
=> (512,)
=> (512,)
=> (3, 3, 512, 512)
=> (512,)
=> (512,)
=> (1, 1, 512, 2048)
=> (2048,)
=> (2048,)
=> (1, 1, 1024, 2048)
=> (2048,)
=> (2048,)
=> (1, 1, 2048, 512)
=> (512,)
=> (512,)
=> (3, 3, 512, 512)
=> (512,)
=> (512,)
=> (1, 1, 512, 2048)
=> (2048,)
=> (2048,)
=> (1, 1, 2048, 512)
=> (512,)
=> (512,)
=> (3, 3, 512, 512)
=> (512,)
=> (512,)
=> (1, 1, 512, 2048)
=> (2048,)
=> (2048,)
=> (1, 1, 256, 256)
=> (256,)
=> (1, 1, 512, 256)
=> (256,)
=> (1, 1, 1024, 256)
=> (256,)
=> (1, 1, 2048, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (1, 1, 256, 3)
=> (3,)
=> (1, 1, 256, 12)
=> (12,)
=> (12544, 1024)
=> (1024,)
=> (1024, 1024)
=> (1024,)
=> (1024, 6)
=> (6,)
=> (1024, 24)
=> (24,)
=> (3, 3, 256, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (3, 3, 256, 256)
=> (256,)
=> (2, 2, 256, 256)
=> (256,)
=> (1, 1, 256, 6)
=> (6,)
%%%%%%%%%%

[MaskRCNN] INFO : # ============================================= #
[MaskRCNN] INFO : Start Training
[MaskRCNN] INFO : # %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% #

[GPU 00] Restoring pretrained weights (265 Tensors)
[MaskRCNN] INFO : Pretrained weights loaded with success…

[MaskRCNN] INFO : Saving checkpoints for 0 into /workspace/tao-experiments/mask_rcnn/exp_unpruned/model.step-0.tlt.
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1365, in _do_call
return fn(*args)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1350, in _run_fn
target_list, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} indices[8] = [8] does not index into param shape [2,3072,4096]
[[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_3629]]
(1) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} indices[8] = [8] does not index into param shape [2,3072,4096]
[[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py”, line 222, in
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py”, line 218, in main
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py”, line 85, in run_executer
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/executer/distributed_executer.py”, line 399, in train_and_eval
File “/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py”, line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py”, line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py”, line 1195, in _train_model_default
saving_listeners)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py”, line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py”, line 754, in run
run_metadata=run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py”, line 1259, in run
run_metadata=run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py”, line 1360, in run
raise six.reraise(*original_exc_info)
File “/usr/local/lib/python3.6/dist-packages/six.py”, line 696, in reraise
raise value
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py”, line 1345, in run
return self._sess.run(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py”, line 1418, in run
run_metadata=run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py”, line 1176, in run
return self._sess.run(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 956, in run
run_metadata_ptr)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1180, in _run
feed_dict_tensor, options, run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1359, in _do_run
run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[8] = [8] does not index into param shape [2,3072,4096]
[[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
[[IteratorGetNext]]
[[IteratorGetNext/_3629]]
(1) Invalid argument: indices[8] = [8] does not index into param shape [2,3072,4096]
[[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
[[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

[MaskRCNN] INFO : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ #
[MaskRCNN] INFO : Training Performance Summary
[MaskRCNN] INFO : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ #
DLL 2021-11-01 12:41:50.709029 - : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ #
DLL 2021-11-01 12:41:50.709336 - : Training Performance Summary
DLL 2021-11-01 12:41:50.709381 - : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ #

DLL 2021-11-01 12:41:50.709432 - Average_throughput : -1.0 samples/sec
DLL 2021-11-01 12:41:50.709481 - Total processed steps : 1
DLL 2021-11-01 12:41:50.709547 - Total_processing_time : 0h 00m 00s
[MaskRCNN] INFO : Average throughput: -1.0 samples/sec
[MaskRCNN] INFO : Total processed steps: 1
[MaskRCNN] INFO : Total processing time: 0h 00m 00s
DLL 2021-11-01 12:41:50.709826 - : ==================== Metrics ====================
[MaskRCNN] INFO : ==================== Metrics ====================

[MaskRCNN] ERROR : Job finished with an uncaught exception: FAILURE
2021-11-01 20:41:55,784 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

gao-yan · November 1, 2021, 1:42pm

seed: 123
use_amp: False
warmup_steps: 1000
checkpoint: “/workspace/tao-experiments/mask_rcnn/pretrained_resnet50/pretrained_instance_segmentation_vresnet50/resnet50.hdf5”
learning_rate_steps: “[1200, 1600, 1800]”
learning_rate_decay_levels: “[0.1, 0.02, 0.01]”
total_steps: 2000
train_batch_size: 1
eval_batch_size: 1
num_steps_per_eval: 100
momentum: 0.9
l2_weight_decay: 0.0001
warmup_learning_rate: 0.0001
init_learning_rate: 0.01

data_config{
image_size: “(832, 1344)”
augment_input_data: True
eval_samples: 68
training_file_pattern: “/workspace/tao-experiments/raw_data/tf-records/train*.tfrecord”
validation_file_pattern: “/workspace/tao-experiments/raw_data/tf-records/val*.tfrecord”
val_json_file: “/workspace/tao-experiments/raw_data/annotations/instances_default.json”
# dataset specific parameters
num_classes: 6
skip_crowd_during_training: True
}

maskrcnn_config {
nlayers: 50
arch: “resnet”
freeze_bn: True
freeze_blocks: “[0,1]”
gt_mask_size: 112
# Region Proposal Network
rpn_positive_overlap: 0.7
rpn_negative_overlap: 0.3
rpn_batch_size_per_im: 256
rpn_fg_fraction: 0.5
rpn_min_size: 0.

# Proposal layer.
batch_size_per_im: 512
fg_fraction: 0.25
fg_thresh: 0.5
bg_thresh_hi: 0.5
bg_thresh_lo: 0.

# Faster-RCNN heads.
fast_rcnn_mlp_head_dim: 1024
bbox_reg_weights: "(10., 10., 5., 5.)"

# Mask-RCNN heads.
include_mask: True
mrcnn_resolution: 28

# training
train_rpn_pre_nms_topn: 2000
train_rpn_post_nms_topn: 1000
train_rpn_nms_threshold: 0.7

# evaluation
test_detections_per_image: 100
test_nms: 0.5
test_rpn_pre_nms_topn: 1000
test_rpn_post_nms_topn: 1000
test_rpn_nms_thresh: 0.7

# model architecture
min_level: 2
max_level: 6
num_scales: 1
aspect_ratios: "[(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)]"
anchor_scale: 8

# localization loss
rpn_box_loss_weight: 1.0
fast_rcnn_box_loss_weight: 1.0
mrcnn_weight_loss_mask: 1.0
}

Morganh · November 1, 2021, 4:38pm

Did you train your own dataset?
Please note that

Input Requirement

Input size : C * W * H (where C = 3, W >= 128, H >= 128 and W, H are multiples of 2^ max_level )

gao-yan · November 1, 2021, 11:38pm

Yes, train my own dataset.
The size of images are different and I set image_size: “(832, 1344)” in the spec file so the H and W both are multiples of 2^6.

gao-yan · November 2, 2021, 1:03am

Solved. Some objects in the images only have bbox without segmentation, so I remove these objects and generate the dataset again. But there is a OOM error when I set train_batch_size = 16 even with 5 Tesla V100 (32GB * 5). So I set train_batch_size = 8, but the training stopping at Saving checkpoints for 0 into /workspace/tao-experiments/mask_rcnn/exp_unpruned/model.step-0.tlt. and only one GPU filled with data. If I only use one GPU, the training is ok but I have to set train_batch_size = 4

Morganh · November 2, 2021, 1:52am

Please remove folder exp_unpruned and mdkir it again.

Can you share the training command?

gao-yan · November 2, 2021, 4:47am

training command as following:

tao mask_rcnn train
-e /workspace/tao-experiments/mask_rcnn/specs/maskrcnn_train_resnet50.txt
-d /workspace/tao-experiments/mask_rcnn/exp_unpruned
-k nvidia_tlt
–gpus 2
–gpu_index 6 7

Morganh · November 2, 2021, 7:27am

So, you are running with 2 gpus.
Please set a lower batch-size if OOM happens.

gao-yan · November 2, 2021, 9:18am

yes, I can use more gpus to avoid OOM, but when I use more than one GPU , the training stops at Saving checkpoints for 0 into /workspace/tao-experiments/mask_rcnn/exp_unpruned/model.step-0.tlt. just at the beginning of the training.

Morganh · November 2, 2021, 9:20am

Please recreate exp_unpruned folder and try again.

gao-yan · November 2, 2021, 11:57am

I remove all files in exp_unpruned and train again. not helpful.
I have to ctrl-C to interrupt the training process because it stops at Saving checkpoints for 0 into /workspace/tao-experiments/mask_rcnn/exp_unpruned/model.step-0.tlt for a very long time and I have no idea if the program is running correctly or not.
Note that, the above problem occurs only when I use multiple gpus.

Morganh · November 8, 2021, 7:53am

Thanks for catching. ~~Unfortunately it is an issue for multi gpus training with Mask_rcnn. We will fix it in next release.~~

Morganh · November 9, 2021, 8:47am

Hi @gao-yan
I want to withdraw my above comment. It is not an issue for the training with multi gpus.
We can run training successfully now.

The root-cause for this issue may be related to your training dataset. If there is only one tfrecord file, please add more tfrecord files in "training_file_pattern: " of your spec.

gao-yan · November 11, 2021, 1:51am

Yeah, You’re right! I collect more data and split the dataset to multiple tfrecords, and now I can use multiple GPUs to train the model. Thanks!

gao-yan · November 11, 2021, 1:57am

Could you give more details about how TAO use multiple GPUs to accelerate the training process and how user like me to use it correctly? Are the image batch split to more mini batches to freed into GPUs?

Morganh · November 11, 2021, 8:46am

Yes, it is data parallelism.
And TAO uses Horovod to implement multi-GPU training. For horovod, please see GitHub - horovod/horovod: Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

system · November 25, 2021, 8:47am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Permission denied: 'mrcnn_log.json' while converting data into tfrecords TAO Toolkit	9	901	August 16, 2022
Can 40 Series RTX products use tao-toolkit? TAO Toolkit	12	907	December 10, 2022
Mask rcnn export needs validation dataset tfrecords TAO Toolkit	23	538	September 18, 2022
Mask_rcnn shows training logs loss 0.00000 fast_rcnn class loss: 0.00000 fast_rcnn box loss: 0.00000 TAO Toolkit	22	805	August 16, 2022
Mask-RCCNN fails on custom dataset TAO Toolkit tensorflow	4	1093	March 4, 2023
FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/getting_started_v5.0.0/notebooks/tao_launcher_starter_kit/mask_rcnn/specs/maskrc TAO Toolkit python	44	1458	September 5, 2023
Transfer learning using the cityscapes dataset in unet-peopleSemSegNet causes poor generalization performance TAO Toolkit segmentation	12	1486	November 10, 2021
Mask_rcnn pruned model inference error TAO Toolkit	6	626	December 5, 2021
Object Detection using TAO DetectNet_v2. The category accuracy results are missing TAO Toolkit	17	660	January 13, 2022
TAO - PIL.Image.DecompressionBombError TAO Toolkit	16	964	December 22, 2023

Train mask-rcnn failure

Input Requirement

Related topics