Hi @betairylia, welcome!
Wow, this is a huge question. It’s a very good question, but I have to be clear from the start that I’m not sure we can give you concrete advice yet. We are definitely happy to discuss some of the tradeoffs of various approaches you might take though.
OptiX (and RTX in general) does indeed have very fast ray-BVH traversal. But keep in mind that RTX has been designed primarily for ray tracing surface representations, and not primarily for volumetric data like voxels or for ray marching. This means that the usual tradeoffs could be even larger for you, and that best practices can vary tremendously depending on exactly what you need.
If you want ray marching through sparse volumes, then one option you should at least consider is Nanovdb. https://developer.nvidia.com/blog/accelerating-openvdb-on-gpus-with-nanovdb/
We have done some performance experiments recently using Nanovdb in combination with a hierarchical OptiX BVH, and it becomes clear that Nanovdb is pretty good at doing exactly what it is designed for – traversing sparse volumes. With a lot of effort, the OptiX representations can sometimes help accelerate traversal of the sparse volume, but not always. Nanovdb does not use RTX BVH traversal, but it is pretty fast because it’s well designed to handle sparse voxel grids. I’d recommend researching a bit and thinking about how to use each API according to it’s strengths - OptiX for ray tracing surfaces and animated objects, and Nanovdb for ray-marching sparse volumes when it makes sense.
To handle AS’s with varying static primitives, do you mean primitives that exist in the accel structure, but sometimes you want to render them, and sometimes you want them to be invisible? We offer a fixed small number of “visibility masks” - currently 8 - that you can use to build a layering system. Visibility masks are implemented in hardware, so they are very fast. If you need more than 8, then you can use any-hit shaders or custom intersection shaders to dynamically decide to skip intersections, however this approach can be considerably slower. You can also dynamically rebuild your IAS’s to include or exclude children, since IAS’s are often rather small. Updating IAS’s results in faster traversal performance than using shaders, at the expense of limited flexibility and the per-frame rebuild. Some OptiX users are mixing all of these methods to get the advantages of each.
If you want to render animated moving objects, that is something where a hybrid renderer that mixes volume techniques with surface techniques may be applicable. In OptiX, you would handle animated meshes using traversable motion transforms: https://raytracing-docs.nvidia.com/optix7/guide/index.html#acceleration_structures#traversable-objects
There are some OptiX users who are mixing Nanovdb with OptiX ray tracing. One way to do this is to setup a scene using an OptiX AS, and inside of it you might have custom primitives that represent volumetic objects. When you hit your custom primitive, you can then launch the ray into the volume object. If the ray misses or exits the volume, you can then continue tracing through the OptiX BVH by re-launching the ray at the volume’s entry or exit point (depending on what you need), or if you’re using any-hit programs to handle the volume, then the ray will continue on it’s own.
I suspect much of this is might be determined more by your game engine requirements than by which technique has the best performance. If you want an asset streaming system to dynamically load and hide nearby partitions of your world, the design of that might add some constraints on what you can do, or lead you more naturally in certain directions.
To answer your specific questions: using a single GAS is probably not recommended for showing and hiding primitives, unless you can take advantage of visibility masks. If you can’t use visibility masks, then it’s going to mean either any-hit shaders or GAS rebuilds. Nested IAS’s is certainly viable, and there’s nothing wrong with it feeling similar to an octtree… it is similar.
I know that doesn’t answer any of your questions completely, but I hope that helps a little.
–
David.