Now we’re talking! Very nice.
Actually I would expect that you can fit more particles into 11GB. (For triangles my very coarse rule of thumb is 10 MTriangles per 1 GB.)
If you’re using the Trvbh builder, look into its chunk settings (acceleration properties) to possibly use less VRAM during build time.
Also splitting into multiple GeometryGroups as mentioned before might help to overcome limits due to VRAM fragmentation.
Assuming the spheres are not ordered in some grid, the striped artifacts in the back of the image are possibly from shadow acne due to self intersections.
You would need to increase the “scene epsilon” value which avoid these self intersections when starting a new ray from a hit surface. Or use a different more robust method to avoid these.
I have not seen that error myself.
I could imagine that the compiler inside OptiX can get confused when not finding a rtReportIntersection after a rtPotentialIntersection.
If you can provide a minimal and complete failing shader code we can take a look.
The attributes themselves must only be calculated between rtPotentialIntersection and rtReportIntersection!
If you forget to declare an attribute which is used in an anyhit or closesthit program, OptiX will report that mismatch.
If you declare and calculate more attributes inside the intersection program than you use inside the anyhit and closest hit programs, OptiX will automatically remove the unused ones as dead code.
If you forget to initialize data in the per ray payload, the resulting error is normally small 8x4 sized screen space rectangles of corruption in your image.
For other debug methods have a look into the OptiX introduction examples. Links here:
I implemented my own “rtAssert” function which uses rtThrow and user exceptions. Search the source code for the define USE_DEBUG_EXCEPTIONS to find some more debug tricks.
Note that enabling exception handling in OptiX impacts the runtime performance dramatically since version 4.0.