How to get the transform matrix of an instance?

[zznewclear13]

Hi, I’m trying to trace the ray against ellipsoids, but having some issue with the normal of ellipsoids.

I set up my scene in this way:

1. Set local aabb for sphere, using its center and radius.
2. Build acceleration structure for one sphere.
3. Build instance acceleration structure for 4 spheres, with transform [1.5, 0, 0, Tx, 0, 1, 0, Ty, 0, 0, 1, Tz]. This spread 4 instance around the box, and scale them by 1.5 along the x axis. The code looks like this :

 float instanceZero[12] = { 1.5, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 3.5 };
 memcpy(optixInstances[0].transform, &instanceZero[0], sizeof(float) * 12);

Then in my __intersection__sphere function, I do sphere-ray intersection using object space ray (optixGetObjectRayOrigin and optixGetObjectRayDirection). Then I get my object space normal float3(p0, p1, p2). I know I need to multiply the inverse transpose of object to world matrix to the object space normal in order to get the actual world space normal for shading. I noticed that OptiX has a function called optixTransformNormalFromObjectToWorldSpace to do that. I tried this:

            float3 transformedNormal = optixTransformNormalFromObjectToWorldSpace(normal);

            unsigned int p0, p1, p2, p3;
            p0 = float_as_int(transformedNormal.x);
            p1 = float_as_int(transformedNormal.y);
            p2 = float_as_int(transformedNormal.z);

But It seemed that this function just multiply an identity matrix to my normal (here is the image, the sharp edge on the ellipsoids comes from shadow, and the blurred edge comes from dot(normal, lightDirection)).

I also tried the optixGetObjectToWorldTransformMatrix function.

            float transform[12];
            optixGetWorldToObjectTransformMatrix(transform);
            //More precisely, I should get the transpose matrix of it.

            unsigned int p0, p1, p2, p3;
            p0 = float_as_int(normal.x * transform[0]);
            //I only scale in x axis and have no rotation,
            //so it is okay to just multiply the first element in transform matrix
            //if transform[0] equals 0.667, I will get correct result.
            p1 = float_as_int(normal.y);
            p2 = float_as_int(normal.z);

But it did not work either.

Here is the image with correct normal.

.

And finally comes my question, is there a way to get my normal correct? I’m wondering if the object to world matrix is not the transform matrix of that instance?

Some other words:
There is one thing that I have noticed: the transform matrix used in optixGetObjectRayOrigin and optixGetObjectRayDirection is exactly what I want. I looked up the documentation and found that:

optixGetObjectRayDirection()
Returns the current object space ray direction based on the current transform stack.

optixGetObjectToWorldTransformMatrix(float m[12])
Returns the object-to-world transformation matrix resulting from the current active transformation list.

I’m wondering if the transform stack and the transformation list are two different things.

[droettger]

If you use an instance transform above a custom sphere primitive, then you do not need to care about the instance transform matrix inside the sphere intersection shader at all because that works in object space.

Just use optixReportIntersection() to return the intersection distance, a specific hitKind value which allows to identify that the hit was on your custom sphere primitive, and the object space normal.

Mind that the hitKind has some reserved value ranges which are used for the built-in triangle and curve primitives.
Explained for optixGetHitKind here:
https://raytracing-docs.nvidia.com/optix7/api/html/group__optix__device__api.html#gaea539824cff7f2f8c3109ce061eb6ffe

When using additional vertex attributes then you need to set the OptixPipelineCompileOptions numAttributeValues to the proper value. Default needs to be two for the triangle barycentrics. You would need three when reporting the sphere normal.

Now everything else happens inside the closesthit program(s).

If you’re using the same closesthit program for different geometric primitives, you need to use the optixGetHitKind() function to determine what primitive type you hit, because the vertex attribute calculation for those might be different.

Then you calculate or read your object space vertex attributes, get the current object-to-world matrix (when needed) and its inverse, the world-to-object matrix, which is used to transform the object space normals into world space.

Depending on the currently active transform hierarchy for that specific hit, getting the concatenated matrices is more or less involved.
The OptiX SDK provides helper functions doing this, which handle the most general case, including motion transform matrix interpolations.
You’ve found them already optixGetWorldToObjectTransformMatrix() and optixGetObjectToWorldTransformMatrix().
If you look at their actual implementation code inside the OptiX SDK, you see how that walks over the transform list.

I’m using those for motion transforms in my simple examples because the necessary calculations are quite involved and require a matrix inversion function because motion transforms don’t hold that, and I didn’t want to duplicate the code to optimize it for that specific simple use case. It’s more expensive than it would need to be in that example.
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/intro_motion_blur/shaders/closesthit.cu#L72

Anyway, that’s effectively the code you need in your own closesthit program!

You do not need the transposed matrix of the inverse object-to-world transformation. That transpose operation is automatically handled when using the correct transform helper function optixTransformNormal() which expects the inverse matrix (aka. world-to-object) as argument and multiplies with the transpose.
If you only ever have just that one normal to transform, there is also the combined helper function optixTransformNormalFromObjectToWorldSpace(). (I’m not a fan of such convenience functions inside an explicit API because that invites misuse.)

Now, for performance reasons you might not want to use these general purpose Get-TransformMatrix implementations.
For example, if you only use a single transform level (OptixPipelineCompileOptions traversableGraphFlags = OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_LEVEL_INSTANCING) then it’s really simple because there is always exactly one instance transform and it holds the inverse as well.

I’ve implemented my own routines for that case here:
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/rtigo3/shaders/closesthit.cu#L46
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/rtigo3/shaders/closesthit.cu#L153

[zznewclear13]

Thanks for your very detailed reply, which really helped me!

It turned out that I wrongly set traversableGraphFlags to OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_GAS and that disabled any transforms.
By changing that to OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_ANY or OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_SINGLE_LEVEL_INSTANCING, I could transform object-space-normal to world space as I did before, or as you mentioned, apply the transformation in closesthit program.

Thanks a lot!