I want to transform my mouse screen coords (x, y) in world coords of my scene.
I have a float3 variable which represents my camera_eye and a float3 variable which represents my camera_lookat. My vfov is 35
So how can i transform my mouse screen cords with these variables in world coords?
First, that is one vector too few to allow that calculation at all. You need one more information to orient the camera roll around the view vector from eye to lookat. That’s normally specified with an up-vector.
Then the OptiX pinhole camera calculates a base coordinate system with eye and U, V, W vectors in world space.
Example image on slide 18 here: http://on-demand.gputechconf.com/gtc/2018/presentation/s8518-an-introduction-to-optix.pdf
More links here: https://devtalk.nvidia.com/default/topic/998546/optix/optix-advanced-samples-on-github/
The 2D mouse coordinates are only 2D so there are infinitely many 3D world positions projecting onto that, means it actually defines some vector.
If you assume the mouse coordinate is on the camera plane spanned by the U and V vectors in length(W) distance, then you can calculate that point easily.
Do not forget to invert the mouse y-coordinate because that has origin at top-left where normal coordinate systems use a lower-left origin.
// Convert integer mouse position at the top-left corner of a pixel into floating point center of the pixel.
float x = float(mouse.x) + 0.5f;
float y = float(mouse.y) + 0.5f;
// Invert origin to bottom-left and normalize mouse position to range [0.0f, 1.0f]
x /= float(windowClientWidth.x);
y = (float(windowClientHeight.y) - y) / float(windowClientHeight.y);
// Scale it to range [-1.0f, 1.0f]
x = x * 2.0f - 1.0f;
y = y * 2.0f - 1.0f;
// Calculate the world position on the camera plane.
// float3 pointOnCameraPlane = cameraEye + x * cameraU + y * cameraV + cameraW;
// Direction from camera eye through that plane point.
// float3 directionMouse = normalize(pointOnCameraPlane - cameraEye);
// Note that the cameraEye cancels out in these two calculations. Means this is the same:
float3 directionMouse = normalize(x * cameraU + y * cameraV + cameraW);
If you shoot a ray from the cameraEye position with that directionMouse, you will hit the 3D world positions the mouse hovers over in your window’s client area (if the whole area defines the projection).
Similarly for any other camera projection than pinhole.