Ok, so there is some sort of GC at work here the figures out that those vertices aren’t needed anymore?
No, all memory handling is your responsibility.
If you hook out an instance from the top-level IAS, the GAS child of that can simply not be reached during traversal anymore (unless it’s still used by another instance). But the GAS memory allocation and traversable handle of that still exist. See below.
this means I rebuild that IAS from scratch, right?
Yes, call optixAccelBuild with all active OptixInstance as build input.
optixAccelBuild fills the CUdeviceptr you provided for the AS and returns the traversable handle of that AS.
That are the two arguments d_outputbuffer and outputHandle in the above API reference link.
These two belong together as long as you use that traversable handle!
You free the d_outputBuffer when you do not need the AS anymore, because that is what you allocated for the AS data.
You must not use that traversable handle anymore after freeing that AS buffer.
No need to “free” that traversable handle, it’s just a 64-bit value which is useless without the CUdeviceptr containing the AS data.