training of FCN_RESNET18_SUNRGB

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi,

I was wondering if the semantic segmentation (segnet FCN_RESNET18_SUNRGB) for the SUNRGBD has been trained only for the RGB part or also for the RGB-D. There are multiple papers, such as https://arxiv.org/pdf/1511.00561.pdf (section 4.2), that mention that the training for segnet or FCN_RESNET18_SUNRGB had been trained only with the RGB part of the dataset.

Can someone confirm this? I am trying to compare segnet from github hello AI world example vs another dnn for specifically the rgb-d images.

Kind regards

2

Hi cmholt1, since FCN-ResNet18 is a segmentation network and does not incorporate depth, it was only trained on the RGB portion of the dataset.

3

Thank you so much for the quick reply!

Do you know of any segmentation network (using cnn) that incorporates depth? and is trained with the rgb-d portion of the set as well?

4

I do not, sorry. Is your goal to produce improved segmentation results from the depth info, or to add depth perception capabilities to the network?

I have been experimenting with mono-depth networks in the ‘depth’ branch of jetson-inference:
https://github.com/dusty-nv/jetson-inference/tree/depth

5

to add perception capabilities to the network. We are comparing CNN vs graph neural network for depth awareness images, so in order to do a proper comparison the model should be trained on rgb-d.

Is the “mono-depth” also doing semantic segmentation?

I really appreciate the quick response!

6

No, they are separate networks. I’m not sure of research papers that combine such capabilities into one network, but perhaps they are out there.

7

Is there a way to find the mean accuracy and the mean IoU? Do you guys have the results? Or is there a python script containing all these metrics?

8

I measure those during training/eval in the PyTorch training script here: https://github.com/dusty-nv/pytorch-segmentation

These are where the accuracy results from this table come from.