BasicWriter and Occlusion for Skeleton generation

I am trying to do SDG for a dataset of skeletons. Currently the only writer which supports Skeleton output is the BasicWriter (OmniWriter). Unfortunately a lot of skeletons in the output come out almost fully occluded or very small (<25px in diameter). Similar issues apply to the output for tight bounding boxes. How can I discard those annotations that are too occluded, if the occlusion annotator always returns -1, because I am generating ActorCore assets that are build from multiple meshes.

What I need is the occluded and size thresholds like in KittiWriter but for BasicWriter but I don’t know how to acheive that.

@pcallender @dennis.lynch

Hi @Turowicz , you can use the bounding box tight to calculate the number of pixel of a prim. The area would just be

(box_tight["x_max"] - box_tight["x_min"]) * (box_tight["y_max"] - box_tight["y_min"])

Based on the number of pixel calculated, you can decide whether or not to write out the data.

For occluded, you can get the ratio between tight and loose bounding box to determine the threashold.

Thanks for the answer but it doesn’t solve the problem. We need to know the % of the full object that is occluded. BB pixel count is only what’s visible, but how we get the total? KittiWriter somehow is able to calculate that for it’s thresholds (partly_occluded, fully_visible)

kitti is using the area ration between loose bounding box and tight bounding box. You can do the same in your custom writer.

That doesn’t make much sense. So the writer doesn’t really know how much of the full object is in fact visible?

@pcallender maybe you can shed some light on this?

Writer doesn’t know the info about the objects. It simply fetches data from annotator, post process them and write them. There is no “direct” info about whether a object is occluded or not, but like I said before, you can compute the occlusion ration by doing a bbox_tight_area / bbox_loose_area. If the ration is greater than 0.95, that’s characterised as fully visible. If it is between 0.5 and 0.95, that’s partial visibly. And below that it is considered occluded. You can set the threshold yourself to meet your needs.

Ok, then I guess what is not clear is what is the meaning of the bbox_loose_area. I assumed it was the tight BB with padding. What is it then? Or better, where is the definition in the documentation?

Found it: Loose bounding boxes bound the entire entity regardless of occlusions

OK here is the problem: What if we can only see a raised hand and a foot? In this situation both bboxes can be be 100% the same but in reality majority of the person will be occluded. We need a better algorithm for this @pcallender @jiehanw @dennis.lynch if you want this tool to be used for serious things.

For example this is fully visible according to the primitive algorithm that is being used by the kitti writer:

What we need is Semantic ID Segmentation LOOSE, to be able to do the math on masks to be more precise with our annotations.

Any chance of achieving Semantic Instance Segmentation Loose? ie. skip occlusions

We really need this guys @pcallender @jiehanw

Hello @Turowicz. Thank you for the feedback. We’re 100% with you and understand how much of an impact the lack of multi-mesh occlusion data can have. The addition of accurate occlusion values for multi-mesh prims has been on our list for some time and we have a plan of how to get there but is still some time away.

In the mean time, given that your dataset has skeletons, if you haven’t already, you may try thresholding on the number of visible joints (jointOcclusions, Annotators Information — Omniverse Extensions latest documentation). Another possibility is to look at certain body parts specifically (body parts less likely to suffer from self occlusion like head and torso to get their accurate occlusion ratio.

These solutions won’t be perfect, so while we’re working on a proper solution in an upcoming release, we’d love to keep working on a current solution with you to unblock your work if these suggestions don’t work in your workflow.

1 Like

Thanks for a workaround but unfortunately I also have non-skeletal objects.

@jlafleche we don’t need multimesh occlusions. We just need “loose” semantic segmentation, that way we will be able to cope.