From the Adding Semantics to a Scene documentation it is clear how to add semantics to primitives as a whole. Though is it possible to have much more fine-grained label semantics (i.e. pixel-level panoptic segmentation) through detailed material/texture metadata that accurately describes those semantics (i.e. label maps)?
This would primarily be for segmenting surfaces that have possibly many detailed concepts on them with different semantics (i.e. textures, etc.).
My general sense is that this might have some relation with NVIDIA’s Material Definition Language and compatible shaders and also the process in Omniverse’s renderer that generates the AOVs. It is unclear, however, how one could add semantics directly to materials (as possibly a separate metadata map) and then have those semantics present in the AOVs for use in an annotator.