As documented in setMatrix function that “If both are valid, the matrices will be used as-is”, but it seems a41, a42, a43 and a44 are reseted to 0, 0, 0, 1 anyway in 6.0. I have some specific matrices where a41, a42, a43 and a44 are assigned. Now it doesn’t work any more. You know, transformation node doesn’t allow any device variables, so I don’t know how to work around this.
It is reproducible in optixInstancing example. You simply need to setMatrix with both matrices, then read a41, a42, a43 and a44 from the device side. In OptiX 5.1 it works, but not in OptiX 6.0.