Train instance segmentation model in google cloud instance with custom data and run inference on Jetson AGX Xavier with TensorRT

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I’m working on a project for my company in which I need to setup a google cloud VM that is going to be used to train models with Transfer Learning Toolkit and then those models need to run on a Jetson AGX Xavier. For now the first model that needs to be trained with custom data is an instance segmentation model. I’ve been trying to find all the resources necessary to achieve these tasks and so far I have:

  1. Setup a google cloud VM with nvidia GCP Image
  2. Train mask r-cnn model following this tutorial until step 4 included (needs to adapt for my VM setup and to load my custom data which is in COCO format).
  3. Move trained model to Jetson AGX Xavier.
  4. Convert model using tlt-converter (adapt code from maskrcnn.ipynb in step 2).
  5. Find python code to make inference with my converted model. maybe from here altought I don’t see examples with instance segmentation.

Am I on the right track here? seems like a lot of work and would really appreciate to not go through this to end up finding out that it wasn’t the right path to follow.

3

Some comments on your steps.
For step 2, under your google VM environment, please make sure Transfer Learning Toolkit — Transfer Learning Toolkit 3.0 documentation is followed. Before you run the jupyter sample notebook, please understand TLT Launcher — Transfer Learning Toolkit 3.0 documentation too.
For step 4, this is one of the methods. Another method is that you can directly deploy the etlt model. Suggest you read MaskRCNN — Transfer Learning Toolkit 3.0 documentation
For step 5, there are sample config files for deploying in deepstream. Please see https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/tree/master/configs/peopleSegNet_tlt and follow GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream . Peoplesegnet is actually Maskrcnn. See PeopleSegNet — Transfer Learning Toolkit 3.0 documentation

4

Thanks a lot for the reply, I’ll do my best to follow all that documentation to tackle the project.