Hi @foteinos, welcome!
The main strategies you can use to achieve your goal are:
1- Break your mesh into multiple pieces, and then build and compact each piece separately. (Use instancing to render the pieces together.)
2- Use less geometry, by culling parts you don’t need, or by using mesh reduction/simplification, or some other mechanism. (I realize this might not be a realistic option.)
How many triangles are in your mesh? Does the size seem like too much for the geometry, or is the main problem figuring out how to render a large geometry on a 4GB GPU?
There are some factors that can affect your memory consumption. You may need extra memory to support
OPTIX_BUILD_FLAG_ALLOW_RANDOM_VERTEX_ACCESS. You will need extra memory to support motion blur. If you’re using motion blur with more than 2 keyframes, you can use fewer keyframes to reduce memory, at the cost of a potentially lower quality blur. There might also be some memory savings if you were to use
The memory usage after compaction is sometimes as much as 2x smaller, but not always. If that is true, then you might be able to fit this mesh into memory with
6.5 / 2 = ~3.25GB, but it’s also possible that the compacted size is close enough to, or greater than, 4GB, which means you won’t be able to render it easily. To state the obvious, of course, the easiest option may be to figure out how to get your hands on a GPU with 8GB or more.
There are also more complicated strategies for rendering when geometry doesn’t fit completely in memory. (I am thinking of this for example: GPU-Motunui · Render blog) However, any such strategy will still require breaking large meshes into individual pieces that can fit on the GPU.