Currently, Mask_rcnn does not support export to ONNX file yet. Suggest you to use trtexec to generate tensorrt engine for running inference on GPU.
Please refer to Converting TAO-trained MaskRCNN models to ONNX for CPU inference - #4 by telo.
Also, there is Mask2Former as alternative. Please refer to latest TAO 5.5 user guide or notebook.