Issue with shadow rays and world vs object coordinates in rtTrace

I’m trying to create a program that uses what would normally constitute rendering rays (back propagation from the camera) and shadow rays.

The rendering rays use the closest hit program while the shadow rays are any hit program.
I found that the closest hit program receives the ray data in world coordinates while the any hit program receives it in object coordinates. Somehow it seems that rtTrace is similarly affected but I’ve not managed to completely figure out how it works. I do see that the any hit program definitely doesn’t get called properly when world coordinates are used for rtTrace.

I am using optic 3.8 and OptixMesh to load the object from an obj file so I’m not sure what code is used for the ray intersection and it seems that OptixMesh is pretty new so may affect the problem.

Anyone else seen the problem and can shed some light?
Is this a bug or a feature and is it documented somewhere how to deal with it?

Thanks

After porting forward ObjLoader, it’s not an issue with Optixamesh

There is definitely something funny going on with the AnyHitProgram vs ClosestHitProgram
When using ClosestHitProgram everything is hit correctly and everything is in world coordinates. With AnyHitProgram intersections are either missing or duplicated, not sure if due to world vs object coordinates or something else, and inside the program everything is in object coordinates in AnyHitProgram while in world coordinates in ClosestHitProgram

I seem to have resolved the issue, not sure if it’s a bug or a feature though, looks more like a bug.

The scene structure is as follows:

(top) Group
-> Transform
-> GeometryGroup

If I set any acceleration structure for the top group other than NoAccel then the any hit program fails to hit a lot of the objects and has duplicate hits in other cases.
This is not a bug with OptixMesh but rather with the traversal of the scene hierarchy.

That the closest hit program sees the rtCurrentRay in world coordinates and the any hit program in object coordinates is correct. Please have a look at the OptiX Programming Guide 4.1.6 Program Variable Transformation.

Group->Transform->GeometryGroup is the standard way to do instancing. To understand what’s going wrong in your last experiment, that would require a little more information, means a reproducer.

Duplicate hits are actually possible in any hit programs because there are acceleration structures in which triangles can be inside multiple BVH nodes. “Sbvh” and “Trbvh” builders are doing that.
If you’re implementing shadow attenuation by counting any hit invocations, that would not work with these two out of the box, try “Bvh” as acceleration builder in that case.
More on this here: https://devtalk.nvidia.com/default/topic/765878/?comment=4327763