I’m not sure I understand what you mean about checking the size of the vertices and triangles before building. Are you using the OptiX mesh sample code to load meshes? Normally, you will have the size of the vertices and index buffers before building, since those sizes need to be given to OptiX to create the buffers. If you’re using the OptiX mesh loader, you can get the mesh size information from the OptiXMesh class members, e.g., getNumVertices(), etc.
Are you using RTX hardware and OptiX 6, or some other combination? Are you using the geometryTriangles API for your meshes, or your own custom format & intersection program? You might experiment and see if using RT_GEOMETRY_FLAG_NO_SPLITTING with rtGeometrySetFlags() or rtGeometryTrianglesSetFlagsPerMaterial() makes any difference to your memory usage.
Your one-third rule of thumb makes sense. It is plausible that with compaction enabled (which is on by default), the temporary high watermark of an Acceleration Structure could be roughly 3x the size of the original mesh data, one copy for your data, one for the BVH, and one for compaction. After the build, usually two of those copies can be freed. With compaction disabled, the third compacted copy wouldn’t be allocated in the first place.
Right now we don’t have a more reliable way in OptiX to estimate the BVH size than what you already figured out. A better way to do that is coming, as Keith mentioned. If 3x the size of your mesh isn’t always working, for now I would suggest trying 3.5x or 4x until it’s always working. If you can break your large mesh(es) into smaller pieces as Keith suggested, you’ll have a lot more temporary space to work with, and be able to fit a lot more geometry on your GPU, since you’re entirely limited by the available free space at any given time. One other idea I can offer is to sort your meshes by size, and load from largest to smallest. That might help if you have a wide variety of sizes.
Also, just to be clear, compaction will only cause a higher watermark if you have a very small number of very large meshes. If you only have one large mesh, then disabling compaction is a good idea. If you have more than a few meshes, then compaction is going to reduce your maximum memory usage because it will compact some of your meshes before building the rest. Generally speaking, if you want to save memory, compaction should be enabled. Can you give some specifics on the number and sizes of meshes you’re trying to load, along with your GPU memory size?