Get world coordinates of the point where mouse clicks in viewport

Isaac Sim Version

4.2.0

Operating System

Ubuntu 22.04

Hey there, I am currently trying to code a ui extension in isaac sim that allows me to enter a selection mode which i can click in the viewport and show me the world coordinates of the point on the surface i clicked on, and possibly spawn a sphere at that location. I’ve seen some discussion about similiar tries, but there are no clear replies.

I’ve also seen the possibility of extracting the way that the Measure Tool uses to select surface points in the viewport for measuring, but i couldn’t understand the code. Any clear guide and help is appreciated!

Hi, any update on this? I was able to use carb to get the position of the mouse click, but I guess this is for the entire UI screen, not the viewport. If it was just viewport, we could do a ray march and get appropriate transforms from camera matrices