I’m encountering a strange issue with rays going through a triangle mesh under certain circumstances. I have a set of triangles in a triangle-strip layout; they are indexed with vertices shared between adjacent triangles. The triangles in question are put into a GAS (as OPTIX_BUILD_INPUT_TYPE_TRIANGLES), and there is an IAS over all GASes. I then raytrace with one ray per pixel for a few frames.
The first time this test runs everything works as expected. A ray might occasionally slip through the geometry between components (the overall model is not watertight), but no rays go through the middle of a mesh. However if multiple test iterations run in a single process, I start seeing rays pass through the mesh. The missed rays are not on the same pixels each time, but only a small fraction of the pixels ever show this issue. The problem pixels produce valid results for the geometry behind the mesh (said geometry is significantly behind the front mesh, not nearly close enough for Z-fighting).
Each test iteration creates a new OptiX context with all new buffers, acceleration structures, etc, and destroys them all at the end. Nothing is saved or re-used. When running with multiple GPUs, each GPU is configured separately. No acceleration structures are copied or moved.
Using OPTIX_DEVICE_CONTEXT_VALIDATION_MODE_ALL seems to “fix” the issue. With it enabled I’ve seen no issues even running hundreds of test iterations, where without it I seldom get to 5 iterations.
Any idea what’s going on or how OPTIX_DEVICE_CONTEXT_VALIDATION_MODE_ALL is changing behavior here? Is it possible/useful to set some specific bits in place of OPTIX_DEVICE_CONTEXT_VALIDATION_MODE_ALL to narrow things down?
I’m testing with OptiX 8.0 and CUDA 12.0, though the issue also appears with OptiX 7.5 and CUDA 11.7. I do not see the issue testing with this geometry (and camera configuration, etc) using OptiX 6.5 and CUDA 10.0, though of course there are numerous code changes between that version and the OptiX 7/8 implementation. I do not currently have a reproducer I can send.
Windows 10 Pro 22H2
dual Quadro RTX 4000 (same issue occurs using one GPU)
Visual Studio 2022