If possible, could you please resize images to smaller ones and retry?
BTW, in Maskrcnn, for 3.0_dp version, see https://forums.developer.nvidia.com/t/tlt-train-maskrcnn-model-with-mapillary-vistas-dataset-failed-on-cuda-error-out-of-memory-out-of-memory , it is recommended to resize to 1/8 or smaller for Mapillary Vistas dataset.