Isaac Sim Version
4.5.0
Operating System
Ubuntu 22.04
GPU Information
- Model:
- Driver Version:
Topic Description
Detailed Description
Essentially, I would like to overlay an image file onto a plane. Generally, I would like to create a material from an image file that I could apply to any object.
Our use case for this is to plot the path of multiple robots and display it on a plane. While a trail is useful in 3D, the 2D image should help with an easy visual. The approach we were taking was to save the plot as a png that will be overwritten during planned intervals, link it to a material in Isaac Sim, and reload the link every frame.
Bonus: if it’s possible to connect a figure earlier and take out the png file middleman, then this could be sped up. How I envision this to work would be directly write to the diffuseTexture using the matplotlib figure. While this isn’t an expected functionality, if this is possible, I would prefer to handle it this way.
Code Snippet
This is our current approach for creating a plane and attaching an image. There’s a bit more to the code we use and I can provide a more complete example if needed, but this is ultimately based on approaches discussed in here ( How can I import a plain PNG image on Isaac Sim? - #4 by fig_kim ). There was a broken link in there.
import isaacsim
from isaacsim import SimulationApp
import omni
simulation_app = SimulationApp({"headless": False})
import omni.usd
from omni.isaac.core.utils.stage import add_reference_to_stage
from pxr import Usd, UsdShade, Sdf, Vt
#NOTE: The World (env) and the Plane are handled by one of our own helper functions
# Create a plane
plane = Plane(name = "Plane", env = env)
plane_prim = env.stage.GetPrimAtPath(plane.prim_path)
# Create a material and assign image
material_path = plane.prim_path + "/Looks/PlaneMaterial"
image_path = "/image/filepath"
material = UsdShade.Material.Define(env.stage, material_path)
# Create a shader
shader = UsdShade.Shader.Define(env.stage, material_path + "/Shader")
shader.CreateIdAttr("UsdPreviewSurface")
shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(0)
shader.CreateInput("metallic", Sdf.ValueTypeNames.Float).Set(0)
shader.CreateInput("ior", Sdf.ValueTypeNames.Float).Set(1.0)
shader.CreateInput("opacity", Sdf.ValueTypeNames.Float).Set(1.0)
color_array = Vt.Vec3fArray([1.0, 1.0, 1.0])
shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Vector3fArray).Set(color_array)
# Connect the shader to the material's surface output
material.GetSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface")
# Add st reader
st_reader = UsdShade.Shader.Define(env.stage, material_path + "/st_reader")
st_reader.CreateIdAttr("UsdPrimvarReader_float2")
st_reader.CreateInput("varname", Sdf.ValueTypeNames.Token).Set("st")
# Add diffuse texture
diffuseTextureSampler = UsdShade.Shader.Define(env.stage, material_path + "/diffuseTexture")
diffuseTextureSampler.CreateIdAttr("UsdUVTexture")
diffuseTextureSampler.CreateInput("file", Sdf.ValueTypeNames.Asset).Set(image_path)
diffuseTextureSampler.CreateInput("st", Sdf.ValueTypeNames.Float2).ConnectToSource(st_reader.ConnectableAPI(), "result")
diffuseTextureSampler.CreateOutput("rgb", Sdf.ValueTypeNames.Float3)
shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).ConnectToSource(diffuseTextureSampler.ConnectableAPI(), "rgb")
# Bind the material the plane
UsdShade.MaterialBindingAPI(plane_prim).Bind(material)
Issue
While there is no direct error message the above, it looks like the image overlaid onto the plane is being sampled from one pixel of the source image rather than using the full image.
I’ve read that this may be due to the lack of a PrimvarReader or UVs not being handled correctly, but the st_reader should be doing the job from my understanding.
Screenshots or Videos
NOTE: When making the post, I had issues with pasting in images. Will append with a comment if they do not show
When using an image that is primarily white.
When using a multi-color image (using Van Gogh’s Starry Night as an example here)
