Trying to get a position where I click in the viewport in the scene

Omniverse Isaac Sim
,
1

Isaac Sim Version

4.5.0

Operating System

Ubuntu 24.04

GPU Information

  • Model: NVIDIA GeForce RTX 3050 Laptop GPU
  • Driver Version: 570.195.03

Trying to get a position in the world by clicking on the viewport

Detailed Description

I am trying to generate an Xform in the world when clicking at some point, generating it at any depth inside the Scene. but it doesn’t matter what I try, I cannot get to transform the mouse 2d coordinates into 3d. My code is the following:

def _screen_to_world(self, screen_coords):
   #screen_cords are the coordinates of the mouse in Isaac editor
   import omni.kit.viewport.utility as vp_util
   import omni.ui.scene as sc

   #first we check if we click inside the viewport

    viewport = vp_util.get_active_viewport()
    window = vp_util.get_active_viewport_window()

    if self.is_on_viewport(screen_coords, viewport, window):
            width, height = viewport.resolution
            mouse_x, mouse_y = screen_coords

            #we transform the mouse coords into viewport coords

            viewport_mouse_x = mouse_x - window.position_x
            viewport_mouse_y = mouse_y - window.position_y

            #we apply formulae to get the ndc (normalized device coordinate) position in  
            ndc_x = (float(viewport_mouse_x)/float(width)) * 2.0 - 1.0 #this is what we get after applying linear equation
            ndc_y = 1.0 - (float(viewport_mouse_y)/float(height)) * 2.0 #for y is different because in x, + is right, while in y, + is up

            to_world_matrix = viewport.ndc_to_world
            carb.log_warn("Conversion: " + str(viewport.map_ndc_to_texture_pixel([ndc_x, ndc_y])))
            #asuming near is -1, we build near and far points

            ndc_near = Gf.Vec4d(ndc_x, ndc_y, -1.0, 1.0) # asuming near = -1
            ndc_far = Gf.Vec4d(ndc_x, ndc_y, 1.0, 1.0) #far = +1
            
            world_near4d = to_world_matrix * ndc_near
            world_far4d = to_world_matrix * ndc_far

            world_near = Gf.Vec3d( world_near4d[0]/world_near4d[3], world_near4d[1]/world_near4d[3], world_near4d[2]/world_near4d[3] )
            world_far = Gf.Vec3d( world_far4d[0]/world_far4d[3], world_far4d[1]/world_far4d[3], world_far4d[2]/world_far4d[3] )

            origin = world_near
            dir = world_far - origin
            dir_norm = dir.GetNormalized()
            
            return origin, dir_norm
    return None, None
2

@squirantes i am just another user and have not tried to do this myself. that said, i do recall an older thread that may or may not be helpful:

3

Hey there, thank you for you answer! Unfortunately, it doesn’t actually says what the person who sent the GitHub Repo says it does. Everything in that extension is made relative to prims in stage and fixed positions. Mouse position is never accessed and never triggered. Is a good extension but not useful for what I need.

What I need is to turn mouse coordinates to world coordinates. The first approach is turn mouse coordinates to ndc ones and turn them into world space using the 4d matrix of the viewport, but it doesn’t work pretty well. That’s why I wanted to know if anyone knew about it.

4

You may find the solution you need in the following thread: Isaac Sim mouse hover mesh highlight. It contains a script example that demonstrates converting mouse coordinates to world coordinates for mesh hover detection in Isaac Sim. This should be adaptable to your click position use case

5

Thanks for your answer. Sadly, this method, even if functional, only retrieves me the origin of coordinates (0,0,0)