Please first follow this search for volume rendering on this OptiX sub-forum and read through the found threads.
That should answer most of your questions already.
Firstly, do I need to make the surface of material as triangles? Or should I write my hit program for voxel? The native voxel mesh has six rectangle surfaces. This seems will increase memory usage by converting to triangle? If I understand correctly, a surface in voxel with 4 vertex will become two triangles with 6 vertex in total?
It really depends on your use case which representation of the volume is best suited.
The ray tracing can only speed up intersections with geometry!
For volume rendering of a regular structure of volume bricks it might make sense to apply standard ray marching instead and track empty and filled volumes with some per-brick flag in a 3D spatial structure (3D buffer or texture).
Maybe even the volume data could be held in a 3D texture and you could ray march through that without any geometry involved.
If you want to generate the regular volume bricks as bounding surfaces, using triangles would be fully accelerated by the RT cores in RTX boards, while a custom box primitive intersection would only have acceleration of the BVH traversal.
Also how to deal with rays that can penetrate the surface layer? Do I add internal vertex also to the AS?
If you have surface geometry which can be penetrated by a ray, you normally reached the closest hit program, which then requires to shoot a new continuation ray on the other side of the surface into the volume for transmission events.
You’d need to track which volume the ray is currently in to handle these transmissions.
If surfaces are bounding volumes with different properties on either side, you would normally track the material (volume) properties of either side of the boundary surface.
Not sure what you mean with internal vertex.
Secondly, if I need to remove some mesh cell during simulation, do I need to rebuild the whole acceleration structure? To clarify, I want to add / remove voxel cells to / from the structure frequently.
If you change geometry you need to rebuild the respective AS.
If this is in some hierarchy of AS and the bounding box of any of the children changed, you also need to update all their parent AS.
If the number of volume cells is huge and you update only few cells, it might make sense to split the while volume into multiple AS spatially (e.g. octants) which would reduce the individual AS size.
Finally, the simulation domain has boundaries. How to deal with rays hit the boundary of domain? Is it possible to apply different boundary conditions?
You mean you have an irregular triangle mesh which encloses the volume?
Then using a ray tracer makes sense to hit that.
Not sure what boundary conditions you mean, but in any case OptiX is just a ray casting SDK and knows nothing about that. All behavior of the rays you shoot are fully under your control. You define the behaviour of “materials” and “volumes” with the device programs your develop