Requested more than 0 entries, but params is empty. Params shape: [0,1920,1080]

While training mask_rcnn on a custom dataset, getting the following error.

[MaskRCNN] INFO    : # ============================================= #                                                                                                                   
[MaskRCNN] INFO    :                  Start Training                                                                                                                                     
[MaskRCNN] INFO    : # %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% #                                                                                                                   
                                                                                                                                                                                         
[GPU 00] Restoring pretrained weights (265 Tensors)                                                                                                                                      
[MaskRCNN] INFO    : Pretrained weights loaded with success...                                                                                                                           
                                                                                                                                                                                         
[MaskRCNN] INFO    : Saving checkpoints for 0 into /mnt/flare/results/model.step-0.tlt.                                                                                                  
Traceback (most recent call last):                                                                                                                                                       
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call                                                                         
    return fn(*args)                                                                                                                                                                     
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn                                                                          
    target_list, run_metadata)                                                                                                                                                           
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun                                                              
    run_metadata)                                                                                                                                                                        
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.                                                                                                     
  (0) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} Requested more than 0 entries, but params is empty.  Params shape: [0,1920,1080]    
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]                                                                                                       
         [[IteratorGetNext]]                                                                                                                                                             
         [[MLP/multilevel_propose_rois/level_2/combined_non_max_suppression/CombinedNonMaxSuppression/_3701]]                                                                            
  (1) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} Requested more than 0 entries, but params is empty.  Params shape: [0,1920,1080]    
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]                                                                                                       
         [[IteratorGetNext]]                                                                                                                                                             
0 successful operations.                                                                                                                                                                 
0 derived errors ignored.                                                                                                                                                                

During handling of the above exception, another exception occurred:                                                                                                                                                
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py", line 222, i
n <module>                                                                                                                                                                                                         
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py", line 218, i
n main                                                                                                                                                                                                             
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py", line 85, in
 run_executer                                                                                                                                                                                                      
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/executer/distributed_executer.
py", line 399, in train_and_eval                                                                                                                                                                                   
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train                                                                                             
    loss = self._train_model(input_fn, hooks, saving_listeners)                                                                                                                                                    
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Requested more than 0 entries, but params is empty.  Params shape: [0,1920,1080] 
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
         [[IteratorGetNext]]
         [[MLP/multilevel_propose_rois/level_2/combined_non_max_suppression/CombinedNonMaxSuppression/_3701]]
  (1) Invalid argument:  Requested more than 0 entries, but params is empty.  Params shape: [0,1920,1080] 
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
         [[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

specs for training

seed: 123
use_amp: False
warmup_steps: 1000
checkpoint: “/home/pretrained_resnet50/pretrained_instance_segmentation_vresnet50/resnet50.hdf5”
learning_rate_steps: “[10000, 15000, 20000]”
learning_rate_decay_levels: “[0.1, 0.02, 0.01]”
total_steps: 25000
train_batch_size: 2
eval_batch_size: 4
num_steps_per_eval: 5000
momentum: 0.9
l2_weight_decay: 0.0001
warmup_learning_rate: 0.0001
init_learning_rate: 0.01

data_config{
image_size: “(832, 1344)”
augment_input_data: True
eval_samples: 500
training_file_pattern: “/home/tfrecords/coco_train*”
validation_file_pattern: “/home/tfrecords/coco_val*”
val_json_file: “/home/val_seg_annotations.json”

# dataset specific parameters
num_classes: 3
skip_crowd_during_training: True

}

maskrcnn_config {
nlayers: 50
arch: “resnet”
freeze_bn: True
freeze_blocks: “[0,1]”
gt_mask_size: 112

# Region Proposal Network
rpn_positive_overlap: 0.7
rpn_negative_overlap: 0.3
rpn_batch_size_per_im: 256
rpn_fg_fraction: 0.5
rpn_min_size: 0.
# Proposal layer.
batch_size_per_im: 512
fg_fraction: 0.25
fg_thresh: 0.5
bg_thresh_hi: 0.5
bg_thresh_lo: 0.

# Faster-RCNN heads.
fast_rcnn_mlp_head_dim: 1024
bbox_reg_weights: "(10., 10., 5., 5.)"

# Mask-RCNN heads.
include_mask: True
mrcnn_resolution: 28

# training
train_rpn_pre_nms_topn: 2000
train_rpn_post_nms_topn: 1000
train_rpn_nms_threshold: 0.7

# evaluation
test_detections_per_image: 100
test_nms: 0.5
test_rpn_pre_nms_topn: 1000
test_rpn_post_nms_topn: 1000
test_rpn_nms_thresh: 0.7

# model architecture
min_level: 2
max_level: 6
num_scales: 1
aspect_ratios: "[(1.0, 1.0), (1.4, 0.7), (0.7, 1.4)]"
anchor_scale: 8

# localization loss
rpn_box_loss_weight: 1.0
fast_rcnn_box_loss_weight: 1.0
mrcnn_weight_loss_mask: 1.0

}

refer : Train Error:tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Requested more than 0 entries, but params is empty. Params shape: [0,1] [[{{node parser/GatherNd}}]] [[IteratorGetNext]] · Issue #156 · google/automl · GitHub

Is it above your own dataset?

Yes. This is for a custom dataset. I have generated tfrecords from the custom dataset

How about running the default jupyter notebook? Is it successful?

Hi Morgan,
I’m not sure about what do you meant by ‘default jupyter notebook’

I have just used the maskrcnn jupyter note book and tao documentation for training the model.

I tried training through terminal only.

I also mean the mask_rcnn jupyter notebook. TAO Toolkit Quick Start Guide — TAO Toolkit 3.21.11 documentation
Before running with your own dataset, did you follow the notebook’s step to train the COCO dataset(mentioned in notebook) successfully?

Hi Morgan , i have tried with default coco dataset, its working . But not working with the custom dataset. Please note the following steps which i used for custom dataset for reference

tfrecord conversion command

python create_coco_tf_record.py --logtostderr --train_image_dir="/usr/Datasets/Segmentation_Dataset/images/" --val_image_dir="/usrDatasets/Segmentation_Dataset/images/"  --
test_image_dir="/usr/Datasets/Segmentation_Dataset/images/"  --train_annotations_file="/usr/Datasets/Segmentation_Dataset/coco_files/train_seg_annotations.json" --val_annotations_file="/usr/Dat
asets/Segmentation_Dataset/coco_files/val_seg_annotations.json" --testdev_annotations_file="/usr/Datasets/Segmentation_Dataset/coco_files/val_seg_annotations.json" --output_dir="/usr/tfrecords/"

output :

Train command :
tao mask_rcnn train -e /usr/specs/maskrcnn_train_resnet50.txt -d /usr/results -k key

Error :

[GPU 00] Restoring pretrained weights (265 Tensors)                                                                                                                                 
[MaskRCNN] INFO    : Pretrained weights loaded with success...                                                                                                                      
                                                                                                                                                                                    
[MaskRCNN] INFO    : Saving checkpoints for 0 into /usr/results/model.step-0.tlt.                                                                                             
Traceback (most recent call last):                                                                                                                                                  
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call                                                                    
    return fn(*args)                                                                                                                                                                
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn                                                                     
    target_list, run_metadata)                                                                                                                                                      
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun                                                         
    run_metadata)                                                                                                                                                                   
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.                                                                                                
  (0) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} Requested more than 0 entries, but params is empty.  Params shape: [0,1]       
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd}}]]                                                                                                    
         [[IteratorGetNext]]                                                                                                                                                        
         [[MLP/multilevel_propose_rois/level_2/combined_non_max_suppression/CombinedNonMaxSuppression/_3701]]                                                                       
  (1) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} Requested more than 0 entries, but params is empty.  Params shape: [0,1]       
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd}}]]                                                                                                    
         [[IteratorGetNext]]                                                                                                                                                        
0 successful operations.                                                                                                                                                            
0 derived errors ignored.

Please try with the comment in Mask R-CNN hangs during training using custom made tfrecords - #20 by Morganh

Is ‘--train_caption_annotations_file’ necessary for creating tfrecords ? This is not mentioned in the official docs
https://docs.nvidia.com/tao/tao-toolkit/text/data_annotation_format.html

It is mentioned in notebook.
If your data doesn’t have caption groundtruth or test set, you can modify download_and_preprocess_coco.sh and create_coco_tf_record.py by commenting out corresponding variables.

create_coco_tf_record.py (11.9 KB)
Commented out lines, but getting error


INFO:tensorflow:Building bounding box index.                                                                          
I0127 14:47:50.553629 139789839779648 create_coco_tf_record.py:212] Building bounding box index.                      
INFO:tensorflow:0 images are missing bboxes.                                                                          
I0127 14:47:50.557328 139789839779648 create_coco_tf_record.py:224] 0 images are missing bboxes.                      
multiprocessing.pool.RemoteTraceback:                                                                                 
"""                                                                                                                   
Traceback (most recent call last):                                                                                    
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker                                              
    result = (True, func(*args, **kwds))                                                                              
  File "create_coco_tf_record.py", line 200, in _pool_create_tf_example                                               
    return create_tf_example(*args)                                                                                   
  File "create_coco_tf_record.py", line 108, in create_tf_example                                                     
    full_path = os.path.join(image_dir, filename)                                                                     
  File "/usr/lib/python3.8/posixpath.py", line 76, in join                                                            
    a = os.fspath(a)                                                                                                  
TypeError: expected str, bytes or os.PathLike object, not dict                                                        
"""                                                                                                                   
                                                                                                                      
The above exception was the direct cause of the following exception:                                                  
                                                                                                                      
Traceback (most recent call last):                                                                                    
  File "create_coco_tf_record.py", line 331, in <module>                                                              
    app.run(main)
  File "/home/ubuntu/.virtualenvs/launcher/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/home/ubuntu/.virtualenvs/launcher/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "create_coco_tf_record.py", line 315, in main 
    _create_tf_record_from_coco_annotations(
  File "create_coco_tf_record.py", line 280, in _create_tf_record_from_coco_annotations
    for idx, (_, tf_example, num_annotations_skipped) in enumerate(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
TypeError: expected str, bytes or os.PathLike object, not dict

Can you run your new create_coco_tf_record.py against COCO dataset?

Getting the same error even with COCO dataset

INFO:tensorflow:Building bounding box index.
I0127 15:11:59.209111 140658956052288 create_coco_tf_record.py:212] Building bounding box index.
INFO:tensorflow:1021 images are missing bboxes.
I0127 15:11:59.720825 140658956052288 create_coco_tf_record.py:223] 1021 images are missing bboxes.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "create_coco_tf_record.py", line 200, in _pool_create_tf_example
    return create_tf_example(*args)
  File "create_coco_tf_record.py", line 108, in create_tf_example
    full_path = os.path.join(image_dir, filename)
  File "/usr/lib/python3.8/posixpath.py", line 76, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not dict
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "create_coco_tf_record.py", line 329, in <module>
    app.run(main)
  File "/home/ubuntu/.virtualenvs/launcher/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/home/ubuntu/.virtualenvs/launcher/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "create_coco_tf_record.py", line 313, in main
    _create_tf_record_from_coco_annotations(
  File "create_coco_tf_record.py", line 278, in _create_tf_record_from_coco_annotations
    for idx, (_, tf_example, num_annotations_skipped) in enumerate(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
TypeError: expected str, bytes or os.PathLike object, not dict

Can you share both commands?

Please try to debug by yourself.

You still need to delete
70 # caption_annotations,

154 #captions = []
155 #for caption_annotation in caption_annotations:
156 # captions.append(caption_annotation[‘caption’].encode(‘utf8’))

171 #‘image/caption’:
172 # dataset_util.bytes_list_feature(captions),

Commented the above lines. Tfrecord conversion is happening with 'include_masks' = False, but getting error while training (Same error mentioned in the top most topic thread).
Getting following error when converting into tfrecords by with include_masks =True

Error

INFO:tensorflow:Building bounding box index.
I0127 16:11:14.873448 140627281082176 create_coco_tf_record.py:209] Building bounding box index.
INFO:tensorflow:0 images are missing bboxes.
I0127 16:11:14.877240 140627281082176 create_coco_tf_record.py:220] 0 images are missing bboxes.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "create_coco_tf_record.py", line 197, in _pool_create_tf_example
    return create_tf_example(*args)
  File "create_coco_tf_record.py", line 143, in create_tf_example
    run_len_encoding = mask.frPyObjects(object_annotations['segmentation'],
KeyError: 'segmentation'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "create_coco_tf_record.py", line 326, in <module>
    app.run(main)
  File "/home/ubuntu/.virtualenvs/launcher/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/home/ubuntu/.virtualenvs/launcher/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "create_coco_tf_record.py", line 310, in main
    _create_tf_record_from_coco_annotations(
  File "create_coco_tf_record.py", line 275, in _create_tf_record_from_coco_annotations
    for idx, (_, tf_example, num_annotations_skipped) in enumerate(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
KeyError: 'segmentation'

It is related to the dataset.
Never mind, you can use the default ‘include_masks’ = False

Created tfrecords by ‘include_masks’ = False. But getting the following error while training !

[MaskRCNN] INFO    : Saving checkpoints for 0 into /usr/results/model.step-0.tlt.                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                 
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call                                                                                                   
    return fn(*args)                                                                                                                                                                                               
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn                                                                                                    
    target_list, run_metadata)                                                                                                                                                                                     
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun                                                                                        
    run_metadata)                                                                                                                                                                                                  
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.                                                                                                                               
  (0) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} Requested more than 0 entries, but params is empty.  Params shape: [0,275,183]                                
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]                                                                                                                                 
         [[IteratorGetNext]]                                                                                                                                                                                       
         [[MLP/multilevel_propose_rois/level_2/combined_non_max_suppression/CombinedNonMaxSuppression/_3701]]                                                                                                      
  (1) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_15633}} Requested more than 0 entries, but params is empty.  Params shape: [0,275,183]                                
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]                                                                                                                                 
         [[IteratorGetNext]]                                                                                                                                                                                       
0 successful operations.                                                                                                                                                                                           
0 derived errors ignored.                                                                                                                                                                                          
                                                                                                                                                                                                                   
During handling of the above exception, another exception occurred:                                                                                                                                                
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py", line 222, i
n <module>                                                                                                                                                                                                         
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py", line 218, i
n main                                                                                                                                                                                                             
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/scripts/train.py", line 85, in
 run_executer                                                                                                                                                                                                      
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/mask_rcnn/executer/distributed_executer.
py", line 399, in train_and_eval                                                                                                                                                                                   
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train                                                                                             
    loss = self._train_model(input_fn, hooks, saving_listeners)                                                                                                                                                    
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model                                                                                     
    return self._train_model_default(input_fn, hooks, saving_listeners)                                                                                                                                            
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default                                                                             
    saving_listeners)                                                                                                                                                                                              
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec                                                                       
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])                                                                                                                                         
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run                                                                                             
    run_metadata=run_metadata)                                                                                                                                                                                     
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run                  
    run_metadata=run_metadata)                                                                                                           
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run                  
    raise six.reraise(*original_exc_info)                                                                                                
  File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Requested more than 0 entries, but params is empty.  Params shape: [0,275,183]
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
         [[IteratorGetNext]]
         [[MLP/multilevel_propose_rois/level_2/combined_non_max_suppression/CombinedNonMaxSuppression/_3701]]
  (1) Invalid argument:  Requested more than 0 entries, but params is empty.  Params shape: [0,275,183]
         [[{{node parser/process_boxes_classes_indices_for_training/GatherNd_2}}]]
         [[IteratorGetNext]]
0 successful operations.
0 derived errors ignored.

[MaskRCNN] INFO    : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ #
[MaskRCNN] INFO    :           Training Performance Summary           
[MaskRCNN] INFO    : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ #
DLL 2022-01-27 16:28:42.677909 -   : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # 
DLL 2022-01-27 16:28:42.678102 -   :           Training Performance Summary            
DLL 2022-01-27 16:28:42.678204 -   : # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ # 

DLL 2022-01-27 16:28:42.678352 -  Average_throughput : -1.0 samples/sec 
DLL 2022-01-27 16:28:42.678416 -  Total processed steps : 1 
DLL 2022-01-27 16:28:42.678486 -  Total_processing_time : 0h 00m 00s 
[MaskRCNN] INFO    : Average throughput: -1.0 samples/sec
[MaskRCNN] INFO    : Total processed steps: 1
[MaskRCNN] INFO    : Total processing time: 0h 00m 00s

Could you refer to Train mask-rcnn failure - #5 by gao-yan ?
That user solved the issue since

Some objects in the images only have bbox without segmentation, so I remove these objects and generate the dataset again.

Checked the custom dataset, Every images mentioned in the json file contains segmentation data.

Is this error related to while converting the data into tfrecords ? Because i didn’t included ‘–include_masks’ flag in the tfrecord conversion command. Was getting ‘KeyError: ‘segmentation’’ when include_masks