First, if you get an out of memory error because you’re constantly rebuilding the acceleration structure, make sure that you correctly manage the underlying CUDA allocations for all CUdevicptr used on those steps.
Otherwise you’re leaking VRAM memory and it would not be surprising that there is an OOM condition after some time.
If you change the matrix inside an instance to transform that underlying IAS or GAS, then the IAS either needs to be rebuild or updated (refitted) before the next launch.
Both is done with the optixAccelBuild() call using different build operations.
You must do that for every acceleration structure which changed and for each IAS which is above any changed acceleration structure up to the root AS of your scene.
If you search the OptiX SDK examples for
OPTIX_BUILD_OPERATION_UPDATE you’ll find examples doing that.
One of my OptiX 7 advanced examples is also showing that inside the intro_motion_blur updateAnimation() function.
AS relocation has nothing to do with that! That only needs to be done whenever you copy or move the CUDA data of an AS pointed to by a CUdeviceptr to another pointer.