Will there be a centralised workflow for creating custom models for semantic segmentation on Jetson devices?

lloydhussell · May 3, 2024, 1:37pm

I would just like to know if Jetson will ever provide a centralised workflow for creating custom models.

I went through the basic guide provided in Dusty’s repository, and that was quite helpful. Where I got kind of stuck was the generation of models and how we then use that custom model.

Is there some place where I can learn about semantic segmentation Jetson devices more in depth? I am happy to pay.

AastaLLL · May 6, 2024, 4:44am

Hi,

Could you share the issue/error on your custom model?

jetson-inference uses TensorRT as a backend inference engine.
So please convert your model into ONNX format and deploy it with TensorRT.

github.com

dusty-nv/jetson-inference/blob/master/docs/segnet-console-2.md#segmenting-images-from-the-command-line

<img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/deep-vision-header.jpg" width="100%">
<p align="right"><sup><a href="detectnet-tracking.md">Back</a> | <a href="segnet-camera-2.md">Next</a> | </sup><a href="../README.md#hello-ai-world"><sup>Contents</sup></a>
<br/>
<sup>Semantic Segmentation</sup></s></p>

# Semantic Segmentation with SegNet
The next deep learning capability we'll cover in this tutorial is **semantic segmentation**.  Semantic segmentation is based on image recognition, except the classifications occur at the pixel level as opposed to the entire image.  This is accomplished by *convolutionalizing* a pre-trained image recognition backbone, which transforms the model into a [Fully Convolutional Network (FCN)](https://arxiv.org/abs/1605.06211) capable of per-pixel labeling.  Especially useful for environmental perception, segmentation yields dense per-pixel classifications of many different potential objects per scene, including scene foregrounds and backgrounds.

<img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/segmentation.jpg">

[`segNet`](../c/segNet.h) accepts as input the 2D image, and outputs a second image with the per-pixel classification mask overlay.  Each pixel of the mask corresponds to the class of object that was classified.  [`segNet`](../c/segNet.h) is available to use from [Python](https://rawgit.com/dusty-nv/jetson-inference/master/docs/html/python/jetson.inference.html#segNet) and [C++](../c/segNet.h).  

As examples of using the `segNet` class, we provide sample programs C++ and Python:

- [`segnet.cpp`](../examples/segnet/segnet.cpp) (C++) 
- [`segnet.py`](../python/examples/segnet.py) (Python) 

These samples are able to segment images, videos, and camera feeds.  For more info about the various types of input/output streams supported, see the [Camera Streaming and Multimedia](aux-streaming.md) page.

See [below](#pretrained-segmentation-models-available) for various pre-trained segmentation models available that use the FCN-ResNet18 network with realtime performance on Jetson.  Models are provided for a variety of environments and subject matter, including urban cities, off-road trails, and indoor office spaces and homes.

This file has been truncated. show original

Thanks.

lloydhussell · May 6, 2024, 7:04am

I have been reading through the tutorial again and I think I follow now.
I need to get my annotated data set, feed it into PyTorch with ONNX as the output format, and then I can provide that model to segNet with the image I want to process.

I will be getting started this week.

kayccc · May 21, 2024, 8:02am

Is this still an issue to support? Any result can be shared?

lloydhussell · May 21, 2024, 8:13am

I still haven’t had the time to get started. But only reading up on the process more.
I do have an idea of how to do it, so that’s fine. However, my original question still stands - will there be a centralised workflow for this from Nvidia at some point?
e.g. a gui application which can take our model annotated images and just do everything with pytorch under the hood, exporting the ONNX format models, to which we can just feed into segnet?