Trying to get a position in the world by clicking on the viewport
Detailed Description
I am trying to generate an Xform in the world when clicking at some point, generating it at any depth inside the Scene. but it doesn’t matter what I try, I cannot get to transform the mouse 2d coordinates into 3d. My code is the following:
def _screen_to_world(self, screen_coords):
#screen_cords are the coordinates of the mouse in Isaac editor
import omni.kit.viewport.utility as vp_util
import omni.ui.scene as sc
#first we check if we click inside the viewport
viewport = vp_util.get_active_viewport()
window = vp_util.get_active_viewport_window()
if self.is_on_viewport(screen_coords, viewport, window):
width, height = viewport.resolution
mouse_x, mouse_y = screen_coords
#we transform the mouse coords into viewport coords
viewport_mouse_x = mouse_x - window.position_x
viewport_mouse_y = mouse_y - window.position_y
#we apply formulae to get the ndc (normalized device coordinate) position in
ndc_x = (float(viewport_mouse_x)/float(width)) * 2.0 - 1.0 #this is what we get after applying linear equation
ndc_y = 1.0 - (float(viewport_mouse_y)/float(height)) * 2.0 #for y is different because in x, + is right, while in y, + is up
to_world_matrix = viewport.ndc_to_world
carb.log_warn("Conversion: " + str(viewport.map_ndc_to_texture_pixel([ndc_x, ndc_y])))
#asuming near is -1, we build near and far points
ndc_near = Gf.Vec4d(ndc_x, ndc_y, -1.0, 1.0) # asuming near = -1
ndc_far = Gf.Vec4d(ndc_x, ndc_y, 1.0, 1.0) #far = +1
world_near4d = to_world_matrix * ndc_near
world_far4d = to_world_matrix * ndc_far
world_near = Gf.Vec3d( world_near4d[0]/world_near4d[3], world_near4d[1]/world_near4d[3], world_near4d[2]/world_near4d[3] )
world_far = Gf.Vec3d( world_far4d[0]/world_far4d[3], world_far4d[1]/world_far4d[3], world_far4d[2]/world_far4d[3] )
origin = world_near
dir = world_far - origin
dir_norm = dir.GetNormalized()
return origin, dir_norm
return None, None
Hey there, thank you for you answer! Unfortunately, it doesn’t actually says what the person who sent the GitHub Repo says it does. Everything in that extension is made relative to prims in stage and fixed positions. Mouse position is never accessed and never triggered. Is a good extension but not useful for what I need.
What I need is to turn mouse coordinates to world coordinates. The first approach is turn mouse coordinates to ndc ones and turn them into world space using the 4d matrix of the viewport, but it doesn’t work pretty well. That’s why I wanted to know if anyone knew about it.
You may find the solution you need in the following thread: Isaac Sim mouse hover mesh highlight. It contains a script example that demonstrates converting mouse coordinates to world coordinates for mesh hover detection in Isaac Sim. This should be adaptable to your click position use case
The issue with your approach is that ndc_to_world gives you a ray (origin + direction) in world space, but not an actual surface hit point — you need to cast that ray against scene geometry to get a 3D position.
There are two clean ways to do this:
Viewport GPU query (recommended for click-to-world-position):
import omni.kit.viewport.utility as vp_util
def on_query_complete(prim_path, world_pos, *args):
if prim_path and world_pos:
print(f"Hit {prim_path} at {world_pos}")
viewport_api = vp_util.get_active_viewport()
# ndc_mouse is the normalized device coordinate of your click (-1 to 1)
pixel, valid = viewport_api.map_ndc_to_texture_ pixel(ndc_mouse)
if valid:
viewport_api.request_query(pixel , on_query_complete)
This uses the renderer’s depth buffer to resolve the world position — no physics setup required.
PhysX raycast (if you need physics-accurate hits):
from omni.physx import get_physx_scene_query_interface
# origin, direction from your NDC unproject (your existing code)
hit = get_physx_scene_query_interface( ).raycast_closest(
carb.Float3(*origin), carb.Float3(*direction), 10000.0
)
if hit["hit"]:
pos = hit["position"] # world-space hit point
print(f"Hit at ({pos.x}, {pos.y}, {pos.z})")
This requires colliders on the geometry you want to hit.
The approach in option 1 is what Kit uses internally for things like focal distance picking. Since this has been idle a while, I’ll close it out — feel free to reopen if you’re still working on this.