Scaling, rotating and translating a ray

I am writing a simple ray tracer using optix 6.5 and I need to apply a different transform to the generated ray for each object. Currently I am generating a transform matrix (S x T x R), but when I apply the matrix to the ray, and perform the ray-object intersection, the objects are scaled correctly, but the rotate does not rotate and the translate, does not translates the objects to the position I expect. I am using optix matrix implementation. Is there something that I am not doing correctly? and is there a better way than manually multiplying the origin and direction of the ray with the tansform matrix.

Maybe start by explaining what the algorithm should solve.

From what you described it sounds like you have a geometry defined in object space and want to move it into world space (per ray) with a transformation matrix build from a scale+translate+rotate operation, most likely in the order v' = S * T * R * v.

But instead of transforming the object space geometry to world space you want to transform some generated ray inversely to produce the appearance as if the geometry is at a defined world space.

That would require the inverse of your per object S+T+R transformation applied to the ray origin and direction.

(Not sure what happens with the scaling in OptiX 6.5 when the ray direction is not normalized.
It might just work because the intersection programs don’t transform the t_min, t_max interval.)

You are correct in what I am trying to achieve. I meant to say that I am currently applying the inverse transform and normalising the direction. What is interesting from your response it the order the matrices, I have been applying them as S * R * T, not as you have specified S * T * R, I know that will make a difference to the look of the geometry. I will try applying the matrices in that order to see if this makes a difference. Would it be better to upgrade to using optix 7.1?

Wait, the order of matrices is what you wrote in the initial post!

Currently I am generating a transform matrix (S x T x R)

The order only depends on what you want to achieve.
Normally you would scale and rotate around the origin and translate last or the pivot changes.
Means I would do v' = T * R * S * v if you want to scale, then rotate, then translate a geometry from object to world space.
(This is all assuming row-major matrices multiplied from the left.)

Now you would just need to figure out in what coordinate space your ray needs to be placed to let the object appear as if it’s in world space.

I always recommend using OptiX 7 versions because the API is a lot more flexible and generally faster overall.
I’m not using OptiX versions before 7.0.0 at all anymore, if that helps your decision.

Thanks for your help. It probably is something to do with the order of the matrices. I think I will start using Optix 7.1