UVW and Right Handed Coord. System

When translating from eye space to world space (or vice-versa) in OptiX, we can use the UVW vectors to build a rotation matrix. My question is (or it’s actually a confirmation I’m seeking), that to create a correct rotation matrix, one has to flip the W vector since the coordinates system has to match.

Normally (in my case too), the UVW vectors build a projection with U right, V top, and W out of the screen. This means that UVW vectors build a left-handed space (matrix). Since all vectors in OptiX are defined in the RHS, we need to flip W to get an RHS matrix.

__device__ void rotateEyeToWorldCoordinates(float3& direction)
{
    const float3 Un =  normalize(U);
    const float3 Vn =  normalize(V);
    const float3 Wn = -normalize(W);

    const float viewMatData[9] = {Un.x, Un.y, Un.z,
                                  Vn.x, Vn.y, Vn.z,
                                  Wn.x, Wn.y, Wn.z};

    const Matrix<3,3> viewMatrix(viewMatData);
    direction = direction * viewMatrix;
}

Even though this might be a really basic question, I would like to get confirmation on this since I might have missed something -> I did not find any flipping in the examples for the pinhole camera which made me wonder.

Yes, the world coordinates inside the examples are defined in right-handed coordinate system and the pinhole camera vectors define a left-handed coordinate system.
Additionally the vertex winding of the triangles is expected to be counter-clockwise inside the provided triangle intersection routines.

Mind that the pinhole camera vectors don’t necessarily need to be orthogonal. That system also allows to define sheared view frusta, which might come in handy for stereo views.

Check this code where I prepared the generation of the normal buffer for the AI Denoiser, which is disabled because the buffer is currently ignored by OptiX 5.1.x versions:
https://github.com/nvpro-samples/optix_advanced_samples/blob/master/src/optixIntroduction/optixIntro_10/shaders/raygeneration.cu#L195

Great thank you very much! :)