I found that there are two opposite definition of instance build input transformation.
In the instance build inputs document, the transformation is defined as A static 3x4 row-major object-to-world matrix transform
However, the API documentation of OptixInstance (https://raytracing-docs.nvidia.com/optix7/api/html/struct_optix_instance.html) and header file of OptixInstance says it is defined as affine world-to-object transformation as 3x4 matrix in row-major layout.
I tested some cases and it seems like the transformation is defined as object-to-world transformation.
Which one is right? or Am I missing something?