Large memory usage

Hi, I’m currently using OptiX 5 on Windows 10 64bit, Geforce 1080 Ti.

I wrote a simple mesh viewer using OptiX. (The structure is almost same as optixMeshViewer example in the SDK. Only some parts(ex. Custom mesh loader, Removed texture support, Single material only) are different.) It works well for 508MB .ply model. But when I put 1.48GB .ply model, I get the following error after calling context->launch().

Memory allocation failed (Details: Function "_rtContextLaunch2D" caught exception: Out of memory)

If I just copy the vertices and faces to optix buffers and do nothing in optix programs, I don’t get this error. But when I create a geometry group for this, set the acceleration structure(Trbvh) and do rtTrace() on that geometry group, I get this error. And sum of the sizes(buffer->getSize() x buffer->getElementSize()) of optix buffers I use is less than 2GB. So my guess is that geometry group or acceleration structure may use a lot of memory… but I’m not sure.

Could you tell me how to solve this problem?

Try to partition the geometry into multiple smaller GeometryGroups holding their acceleration structures (AS).
The Trbvh builder needs quite some amount of memory during AS building.
The acceleration properties for Trbvh allow to build the AS in chunks.

Also see this thread:

Then there is a newer OptiX 6.0.0 version available which changed many things and is normally faster than OptiX 5.1.0. Read all recent forum posts about it.

Also, a few things to keep in mind:

  • How much GPU memory is being used outside of optix? You can determine this by running nvidia-smi before starting your optix app
  • What other memory are your using in optix beside this? output buffers, material index buffers, etc?