Error in InstanceImage segmentation

Autonomous Machines Isaac Omniverse Isaac Sim
#1

Hi all,

So my group is trying to get class-wise and instance-wise segmentation masks from images in a simulated environment of our own creation.

In this environment we have a bunch of objects which already had a class label in their object label (e.g. “chair”) and we gave a printer in the image a class label by setting IsaacSimTag for the object to “printer”.

We captured the class-wise and instance-wise segmentation masks across several images and found some odd results. The attached file seg_class.png shows the class segmentation image attained for a given moment in time. Note the yellow printer in the background and the chair to the far right of the image.

Now this all looks correct but when we go to the instance-wise segmentation mask we get the following image (see instance.png file attached).

Here, the printer and the chair to the far right of the image have been given the same instance id (colour) as the wall. They have not been segmented correctly.

However, at a different frame, we managed to get an instance segmentation for the chair on the right when looking at it from a different angle. (see instance_2.png file attached)

Is this a bug or can someone explain what is going on here and how to fix it?
seg_class.png
instance.png
instance_2.png

#2

Hi d20,

I can not see the images, would you please add or attach them again.

Thanks,
Liila

#3

I have updated the post to now have the images as downloaded attachments.

#4

Hi d20,

Having the same instance number is not a bug, here is an example:

printer class = 1 instances will be 1, 2, 3, …
chair class = 2 instances will be again 1, 2, 3, 4, …

But the pixel value should be different, because the pixel value is a combination of the class and instance.
So it’s ok if the instance colors are the same.

But the colors shouldn’t be different in different frames, that we are looking into into it to make sure is not a bug.

Hope this helps,
Liila

#5

Hi Liila,

Thanks for getting back to me. I am confused by what you are saying based on what I have seen.

I understand that printer class can have instance ids 1,2,3,4 and chair class can also have instance ids of 1,2,3,4.

However when you say “the pixel values should be different, because the pixel value is a combination of the class and instance” alongside “it’s ok if the instance colours are the same” I have gotten confused.

In my images the colour is representative of the pixel value in the InstanceImage. In the InstanceImage the pixel values are the same and you cannot separate the chair from the wall using only the pixel values.

Based on what you had said before is it expected that these should be the same or should they be different?

Is the InstanceImage only supposed to be used in conjunction with the LabelImage so that we can know how to separate individual instances?

#6

Sorry if I confused you, let me explain:

In the instance image that you see the colors are based on the instance value, and are for visualization only. It means that the class images is only based on class numbers ,and the instance image is only based on instance number.

But in the segmentation image that is sent to your algorithm, the RGB pixel value is as follows: R: class number GB: instance number.

Hope this helps,
Liila

#7

Hi Liila,

Sorry but still confused a little.

We are currently only using data output from a SegmentationCameraProto which has components instanceImage and labelImage which are what I have referred to throughout. We are just using Matlab to visualise the output images from this SegmentationCameraProto.

From my understanding the output from the SegmentationCameraProto is what you are referring to in your most recent responses about class images and instance images.

What is the segmentation image that you refer to that you say gives the class number in R and the instance number in GB? What type of Proto is that image being sent along and what codelet translates from the raw SegmentationCameraProto to provide this image?

#8

Hi d20,

Sorry for the delay.
One of our main developers, will provide you precise answer today.

Thanks for your patience,
Liila

#9

Hi d20,

I’m attaching a few images from the office map where I have segmented two classes, the sofas and the rope barriers

On the left side you see the label image: The first class is red and the second class is green.
On the right side you see the instance image: Instances are colored as follows:

1 - red
2 - green
3 - blue
4 - yellow
5 - cyan
6 - orange
7 - pink
8 - magenta

Both the rope barriers and the sofas have matching instance numbers, the first instance for each is red, and then green and so on. Numbering is determined by the order the objects were added to the scene and how UE4 decides to order them in its internal data structures.

In order to get a unique segmentation, both the label and instance values need to be used and there will be a unique instance value per class but different instance values can match each other for the same class

I’ve also attached a video that shows the instance and labels remaining stable across frames. Camera framerate is low to keep the video size small.

Label_instance_sofa_barrier.png
Label_instance_sofa_barrier.png3440×1440 706 KB

IsaacSim_Label_Instance_Sofa_Barrier.avi (2.53 MB)

#10

Thanks for the response, that is what I was starting to suspect.

Are there any tools already in place to use the two segmentation images together in this way or will I just write it up myself?

It is no real issue to make it myself but I want to make sure there isn’t already something available.

#11

There aren’t any tools at the moment sorry, but I’m happy to help if you had any specific questions about how to combine the two images.

#12

Thanks for the offer. If I have any further questions I will post them here.

Any ideas as to what may have happened in our simulator situation where the instance ID changed between viewpoints? (part of the original post).

We haven’t been back to this issue yet in our lab so I thought I would check if you had any thoughts before starting work on code using the two segmentation images together.

#13

Hi d20,

yes, that was the main issue for us (situation where the instance ID changed between viewpoints).
That shouldn’t happen and we did some investigation and couldn’t repro it!
Please let us know if it happened to you again.

Thanks,
Liila