In the OptiX 7.0 Documentation (5.2), I see we need to provide a 3x4 object-to-world matrix for an instance build input. However, in the source code of class OptixInstance, I found the annotation about the variable transform (line 464 in optix_7_types.h) is world-to-object.
Hi @dhart, I find that setting the variable transform of IAS build input using the object-to-world matrix is correct.
Here is my simple test:
First, I make the camera look at the center of the scene's AABB. I construct the AABB according to the local mesh transformed by object-to-world matrix.
Then, while running my program, I can see my scene only if I set the variable transform as the object-to-world matrix instead of the world-to-object one.
And I find what sutil/Scene.cpp does is also to set that variable as an object-to-world matrix.
So, the annotation of the header file seems wrong.
By the way, I find another mistake of the header file sutil/Aabb.h (line 31):
Dang, I guessed wrong. Usually the header files are more current than the docs. Good catch on the include as well. Thank you for pointing these mistakes out, the feedback is helpful!