training of FCN_RESNET18_SUNRGB


I was wondering if the semantic segmentation (segnet FCN_RESNET18_SUNRGB) for the SUNRGBD has been trained only for the RGB part or also for the RGB-D. There are multiple papers, such as (section 4.2), that mention that the training for segnet or FCN_RESNET18_SUNRGB had been trained only with the RGB part of the dataset.

Can someone confirm this? I am trying to compare segnet from github hello AI world example vs another dnn for specifically the rgb-d images.

Kind regards

Hi cmholt1, since FCN-ResNet18 is a segmentation network and does not incorporate depth, it was only trained on the RGB portion of the dataset.

Thank you so much for the quick reply!

Do you know of any segmentation network (using cnn) that incorporates depth? and is trained with the rgb-d portion of the set as well?

I do not, sorry. Is your goal to produce improved segmentation results from the depth info, or to add depth perception capabilities to the network?

I have been experimenting with mono-depth networks in the ‘depth’ branch of jetson-inference:

to add perception capabilities to the network. We are comparing CNN vs graph neural network for depth awareness images, so in order to do a proper comparison the model should be trained on rgb-d.

Is the “mono-depth” also doing semantic segmentation?

I really appreciate the quick response!

No, they are separate networks. I’m not sure of research papers that combine such capabilities into one network, but perhaps they are out there.

Is there a way to find the mean accuracy and the mean IoU? Do you guys have the results? Or is there a python script containing all these metrics?

I measure those during training/eval in the PyTorch training script here:

These are where the accuracy results from this table come from.