Future plans for instance segmentation model in tao toolkit

[laerke.noerregaardhansen]

I have a camera in a fitted environment where I need:

• An edge-friendly model
• Instance segmentation output

At GTC last year, I was informed that a new implementation of Mask R-CNN would be added to TAO Toolkit and that class_weight would be introduced in the configuration. Now, a year later, I have the following questions:

1. Are there any plans to add a new implementation of Mask R-CNN or a more edge-friendly instance segmentation model to TAO Toolkit (e.g., YOLOv8-Seg)?
2. Would it be possible to include class_weight/weighted loss functions in the configuration?
3. Are there any plans to introduce an instance segmentation solution for BYOM (Bring Your Own Model)?

[Morganh]

@laerke.noerregaardhansen

a new implementation of Mask R-CNN or a more edge-friendly instance segmentation model to TAO Toolkit

There is mask2former. You can refer to https://docs.nvidia.com/tao/tao-toolkit/text/cv_finetuning/pytorch/instance_segmentation/mask2former.html.

@laerke.noerregaardhansen

• Would it be possible to include class_weight/weighted loss functions in the configuration?

No, the Mask R-CNN configuration does not currently support direct class_weight parameters in its loss configuration.

@laerke.noerregaardhansen

Are there any plans to introduce an instance segmentation solution for BYOM (Bring Your Own Model)?

It is not in the plan. And actually in TAO 5.0.0, BYOM with TF1 (Classification and UNet) has been deprecated because the source code of TAO is now fully open-sourced. The info is from https://docs.nvidia.com/tao/tao-toolkit/text/byom/index.html.