How to read .engine model's mask segmentation output from `tao mask_rcnn inference` command?

I’ve successfully trained and exported my custom mask-rcnn model using the TAO Toolkit.

Using the tao mask_rcnn inference command on the .engine file I can output the prediction information to json files. All of the json fields are readable except for the segmentation mask which is encoded. How do I converted the encoded segmentation data back to numerical values?

Example Json entry:

{‘area’: 19812,
‘bbox’: [278, 1002, 254, 78],
‘category_id’: 1,
‘id’: 0,
‘image_id’: ‘frame_003740.PNG’,
‘is_crowd’: 0,
‘score’: 0.83984375,
‘segmentation’: {‘counts’: ‘fWV97aQ12M2O001N101O001O001O1N101O0O010O100O1O01000O10000000000O1000000O2O00000O1000000DWOhoNj0XP1VOhoNj0WP1WOioNi0VP1YOhoNh0TP1\OkoNe0TP1\OloNd0TP1\OkoNe0TP1\OloNd0TP1]OkoNc0UP1]OkoNc0UP1]OkoNc0TP1^OkoNc0UP1^OioNc0VP100001O000000O100000000O10000O11O0O2O00001O0000000000000001N1000000000O1000001O0000000000000000000000000O10OO2N2O10O1000000000000O10O100000000000001O00001O00000000O100000O10UOQPOLPP14PPOLPP14PPOLQP13ooNMQP13ooNMRP12noNNRP12noNNRP12noNNSP11moNOSP1k0000VOmoNOSP10ooNOQP11ooN[OO>RP16QPOZON0QP16QPOZON0UP11TPONlo01UPOOko00VPO0jo0OXPO0ho01VPO0jo00VPO0jo00VPO0jo01UPOOko01TPO0lo00loN_O3a0QP10loNA10RP1OloNB2?RP1OloNB2?RP1OloNB2>RP11loNA2>QP12noN_O1?QP12noN^O20PP13UPOMjo04VPOLjo04VPOLjo04VPOLio05ZPOHfo08ZPOHfo08ZPOHfo08ZPOGgo08ZPOHfo08ZPOHfo08[POGeo08\\POHdo08\\POHdo08\\POHdo08\\PO_O@4TP1<]PO@_O4TP1<]PO@_O4TP1<]PO@@3SP1=^PO^O@4SP1>]PO^O@4SP1>]PO^O@4SP1>]PO^O@4SP1>]PO]OB3RP10\PO^OA3RP1?]PO_O_O3TP10ZPOFfo0;YPOEgo0?ooNWOO;QP10moNVO2:QP1f0moN[OSP1e0moN[OSP1f0loNZOTP1f0loNZOUP1d0loN\OTP1d0loN\OTP1d0loN\OTP1d0moNZOTP1e0moN[OSP1d0noN\OSP1c0moNUON1UP1i0ooNTOZP1l0foNSO[P1l0900L40WoNZOP1b0\\oN]O33P1?_oN]O14P1?_oN]O14P1>oN^O05_P1<hoNDWP1=joNAWP1?ioNAWP1?ioNAWP1?joN@VP10koN_OUP10loN@TP10moN_OSP1a0moN_ORP1b0noN^ORP1a0ooN_OQP1a0ooN_OQP1<_oNF0NQP1<oND00PP1<TPODlo0<TPODlo0;UPOEjo0<VPODA0oo0<POD@1QP19POF@0PP12hoN4\\Q1KfnN4ZQ1KhnN3Q1MTYn]1’,
‘size’: [1080, 1920]}}

While running inference via MaskRCNN — TAO Toolkit 3.22.02 documentation, there are output annotated images.

  • -o, --out_image_path : The directory path to output annotated images.

Thanks for the reply. I had already output the annotated images but I needed the isolated segmentation predictions for another piece of the project I am working on.

For anyone else encountering this road block:

The segmentation output is encoded in COCO compressed RLE (Run-length encoding) format (see here: cocoapi/mask.py at master · cocodataset/cocoapi · GitHub).

To go from the encoded string to segmentation contours do the following:

import pycocotools.mask as mask
import json
import cv2
import numpy as np

json_dir = “./JSON-DIR”
f = open(json_dir+’/prediction_json-file.json’,‘r’)
annos = json.load(f)
f.close()

seg_obj = annos[0][‘segmentation’]
mask_obj = mask.decode(seg_obj)

def polygonFromMask(maskedArr):
# adapted from https://github.com/hazirbas/coco-json-converter/blob/master/generate_coco_json.py
contours, _ = cv2.findContours(maskedArr, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
segmentation = [ ]
valid_poly = 0
for contour in contours:
#Valid polygons have >= 6 coordinates (3 points)’
if contour.size >= 6:
segmentation.append(contour.astype(float).flatten().tolist())
valid_poly += 1
if valid_poly == 0:
raise ValueError
return segmentation

poly_obj = polygonFromMask(mask_obj)

This will result in potentially several polygons per prediction so make sure to use all of them to match the original model output

Hope this will help someone!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.