Optix PathTracer: how to implement an updateGeometry functionality

First, when you want to update the render graph (Instance and Geometry Acceleration Structures (IAS, GAS)) inside an OptiX application you need to call optixAccelBuild for all ASes which changed.

I don’t like explaining that for the optixPathTracer example inside the SDK because that is only using a hardcoded geometry acceleration structure which isn’t really what you want when adding objects to a scene.

The better render graph structure for that is a two-level hierarchy with a top-level IAS with OptixInstances which reference GAS. That will also immediately allow geometry reuse by instancing.

Acceleration structure rebuilds or updates need to happen as follows:

  1. When you change the topology of the geometry, that is, changing the number of the vertices or changing any of the indices in an indexed mesh, you must rebuild that GAS.
    That will produce a new traversable handle which you need to put into the OptixInstance above it.

  2. When you only change the vertex positions inside a geometry (like when morphing a mesh), you also must call optixAccelBuild but it can be either a rebuild or an update operation.
    Updating (“refitting”) AS requires that the initial AS was built with the OPTIX_BUILD_FLAG_ALLOW_UPDATE flag.
    https://raytracing-docs.nvidia.com/optix8/guide/index.html#acceleration_structures#dynamic-updates
    But there can be cases where updates degrade the performance of the AS traversal and a full rebuild would improve that again.
    Explained here: https://forums.developer.nvidia.com/t/updating-as-leads-to-extremely-low-traversal-performance/267416

  3. When any GAS inside the render graph changes, all IAS inside the graph which can reach the changed GAS must be rebuilt or updated with optixAccelBuild as well.
    That is necessary because any change to the AABBs of the lower AS must be reflected inside the BVH.
    With a two-level IAS->GAS structure that means you must call optixAccelBuild on the top-level IAS everytime anything changes inside the GASes attached to it. Building IAS is very quick.

  4. When changing transform matrices inside an OptixInstance inside an IAS, the IAS must be rebuilt or updated.
    All IAS above that changed IAS must also be rebuilt or updated since the AABBs changed.

  5. When the number of OptixInstance inside an IAS changes, that is, when adding or removing instances, you must rebuild the whole IAS. Update won’t work because the scene topology changed.
    All IAS above that changed IAS must also be rebuilt or updated since the AABBs changed.

  6. When using motion transforms (linear or SRT) inside the render graph and these change transforms, all IAS above them must be rebuilt or updated. Usually update is fine because only the AABBs of the children changed, though with motion AS that can become more involved.
    Shown in this example: https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/intro_motion_blur/src/Application.cpp#L2490

Note that there is also the optixDynamicGeometry example inside the OptiX SDK which shows this.

Now with all that said, if you still want to use the optixPathTracer as basis for your experiments, updating the GAS inside that would require code changes in various places, because all the geometry and the material assignments to the individual primitives are hardcoded and only work together.

Means you wouldn’t only need to change the g_vertices array but also the g_mat_indices. These should all become non-const and put into std::vectors when you plan to update them.
If you want to add more materials dynamically, everything based on the MAT_COUNT would also need to be made dynamic, that includes the SBT.

Then you would need to call buildMeshAccel( state ); with the updated data. But that function is not meant to be called more than once inside the current code and must be changed to allow dynamic memory allocations, means it needs to be able to free and alloc the state.d_vertices and state.d_gas_output_buffer each time it’s called and not only at program shutdown.

Note that the program is limited to the four hardcocded materials MAT_COUNT but that means you wouldn’t need to change the SBT size, but you must update the hitgroup_records additional data. Search the code for data.vertices.
That happens inside createSBT() which again is not meant to be called more than once inside that application. You would again need to change that or build an update version of that function which handles dynamic free and alloc changes and updates every time you change the geometry data.

I would still recommend implementing an IAS->GAS scene structure where you can more easily update the scene by adding or removing complete GAS. If each of them has a single SBT record, updating the SBT accordingly isn’t difficult either and the IAS sbtOffset is basically selecting the material shader with this formula: https://raytracing-docs.nvidia.com/optix8/guide/index.html#shader_binding_table#accelstruct-sbt

Related thread with AS and SBT tips and tricks: https://forums.developer.nvidia.com/t/question-about-instance-acceleartion-struction/283898/4