[Resolved] Crash on call to rtGeometryTrianglesSetVertices

I’m experiencing crashes from calls to rtGeometryTrianglesSetVertices. In the verbose output I see the following text, but then the program exits unexpectedly with no error message and a non-zero return value.

INFO: Program cache HIT  : aabb_indexed_100_000
INFO: Program cache HIT  : intersect_indexed_100_000

Sometimes I can get the program to run if I delete the OptixCache directory, but this does not always work.

I am sending traces to the help email.

The environment is RTX 8000, Windows driver 418.81, nvcc version 10.0.130


There is a third behavior that occurs sometimes, which is a failure with the following error message:

Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Assertion failed: "static_cast<size_t>( recordIndex ) < m_sbt->size()", file: <internal>, line: 1476)

Both of these errors arise from running the same code without any modifications to the input. Unfortunately, I can’t find anything deterministic about this or whether the two errors are related.

This sure smells like a memory error. Have you sanity checked your geometry-triangles input (including strides and offsets)? Can you reproduce with a very small scene (eg, single triangle or tet)?

Hi Keith,
It does smell like a memory issue. I’ll try with a single triangle scene tonight to see if it reproduces.

Regarding stride an offset, I currently have the rtGeometryTrianglesSetVertices stride set to sizeof(float3) and rtGeometryTrianglesSetTriangleIndices stride set to sizeof(uint3). I previously had them at zero, but received an error. Is this expected behavior, or could it be symptomatic of a memory issue, too?

Is there a sample that shows RTgeometrytriangles used with per-triangle material assignments?

That setting of strides looks correct. You can also look at optixpp_namespace.h for an example of how to set strides introspectively (via Buffer::getElementSize() ).

As for per-tri material example, the optix SDK contains a class OptiXMesh in sutil that sets per-triangle materials assignments.


It seems like it was a memory issue. The problem went away after some code refactoring.

Glad to hear!