We need to train our AI to detect contaminant defects on our product that can be underneath our transparent label. We want to be able to segment these without the label getting in the way. We know that we could use the cutout_opacity parameter of the material’s geometry to completely hide parts of the label, but this as photorealistic as parts of the label being transmissive. I have included a screenshot of this as an example. One way we thought of to fix this was to set only that part of the label with content to have semantics and to make the rest “invisible” to semantics in a way. Is this possible in MDL? We also tried setting the “invisibleToSecondaryRays” and “doNotCastShadows” flags, but these just make the entire label completely invisible in the segmentation. Is this intended behavior and, if so, is there an elegant way to hide only transparent areas in the segmentation? Our current, albeit hacky, solution is to step Orchestrator twice to get the segmentation first and the RGB second. If this is the best solution, is there at least a way that we could disable the RGB or the segmentation render? To my knowledge, even when we only use the RGB output, the segmentation view will still get rendered and vice versa. Another solution we thought of was to separate our label into two meshes, one containing just the content and the other containing everything else, but it would be nice if we could leave the label in one piece. One final question: is there any way to make an object visible only to secondary rays and invisible to all others?
MDL materials can have a semantic, but you’d still run into the issue of needing to make the rest of the label material invisible to the semantic mask, which currently is controlled by the “doNotCastShadows” on the geometry as you mentioned.
“doNotCastShadows” to remove semantic consideration is intended behaviour, and used for cases like window panes where you want the semantic information, because the RGB basically shows a mostly transparent window.
You’ve hit on the ultimate fallback of running the orchestrator twice. I’ll check with the engineers on whether we can avoid the rendering or one or the other.
The separation of the defect is also another approach we essentially use with the material projection technique. Under the hood the node creates a geometry duplicate with a slight perturbing of the mesh normals outward and we drop a material on it to represent the defected parts. Since its a separate geo, we can handle the segmentation as needed.
I’ll check on making an object visible only to secondary. If I’m understanding you correctly, you want to be able to segment something thats invisible in the RGB, so you can use it as a proxy for segmentation?
“doNotCastShadows” to remove semantic consideration is intended behaviour, and used for cases like window panes where you want the semantic information, because the RGB basically shows a mostly transparent window.
Good to know! Is there some way we could assign “doNotCastShadows” to one part of the mesh and not another or some part of a material and not another? For example, make the transparent part of our label have “doNotCastShadows” selected and the content of our label have it unselected thereby making only the content/text segmented and still allowing for the transparent look of the label rather than using “cutout_opacity”.
You’ve hit on the ultimate fallback of running the orchestrator twice. I’ll check with the engineers on whether we can avoid the rendering or one or the other.
That would be awesome! We have always been wondering if there are hidden tools available with more control.
The separation of the defect is also another approach we essentially use with the material projection technique. Under the hood the node creates a geometry duplicate with a slight perturbing of the mesh normals outward and we drop a material on it to represent the defected parts. Since its a separate geo, we can handle the segmentation as needed.
Exactly! We looked at your code for inspiration! A lot of our defects have separate geometry for the sole purpose of having a different segmentation. It would be extremely useful if we could have a single mesh but the material would define different semantics on different parts of it. Correct me if I’m wrong, but I think this would make rendering a lot faster due to fewer collisions during path tracing.
I’ll check on making an object visible only to secondary. If I’m understanding you correctly, you want to be able to segment something thats invisible in the RGB, so you can use it as a proxy for segmentation?
Yes, I want to make it invisible to the RGB but visible to the segmentation. An example of this is that we want to segment what is missing (and train an AI to detect that part of the product is missing). Currently we do this by, again, two-stage rendering which is overly complicated and something we’re not super happy with.
Something else we tried which we though would work was to make a custom MDL that was invisible in all aspects (it doesn’t refract or reflect) but with the “doNotCastShadows” attribute set to False so it would be segmented by the secondary rays. Unfortunately, the ray tracer seemingly still interacted with it and it could be faintly seen and–big surprise–it cast shadows.
I’d love to get your thoughts!
Is there some way we could assign “doNotCastShadows” to one part of the mesh and not another or some part of a material and not another?
Not to my knowledge. “doNotCastShadows” is a mesh prim attribute. I’ll see if I can pull over the MDL experts’ ear see if they know more.
I checked on the ability to render only segmentation, without the heft of the RGB. Currently no. This particular ask came up for a different use case, so we’re looking into it. If we do address this, it wont be soon though.
It would be extremely useful if we could have a single mesh but the material would define different semantics on different parts of it. Correct me if I’m wrong, but I think this would make rendering a lot faster due to fewer collisions during path tracing.
Likely, but it depends a lot of the content of the scene how much. The whole thing was a little bit of a hack that evolved from a demo test I did for something we needed on the spot, and we decided to share until others could address the core need properly.
Yes, I want to make it invisible to the RGB but visible to the segmentation.
My current findings say no, but still digging.
Something else we tried which we though would work was to make a custom MDL that was invisible in all aspects (it doesn’t refract or reflect) but with the “doNotCastShadows” attribute set to False so it would be segmented by the secondary rays. Unfortunately, the ray tracer seemingly still interacted with it and it could be faintly seen and–big surprise–it cast shadows.
I did exactly the same thing in a test a while back. There was always some artifacting from the transparent material and yes … the shadows.
I think the double orchestrator approach, or some procedural projection material approach is the surest way for now. There’s also multi-matte, but its completely outside the SDG pipelining, but I thought I’d mention it here incase it sparks some ideas.
Thank you for taking the time to look into this for us! We’re looking forward to hearing what the MDL experts have to say and if you discover anything else in regards to a mesh being visible to segmentation but invisible to RGB.