Please always provide the following system configuration information when asking about OptiX issues:
OS version, installed GPU(s), VRAM amount, display driver version, OptiX (major.minor.micro) version, CUDA toolkit version (major.minor) used to generate the input PTX, host compiler version.
Using cuda-gdb implies a Linux OS.
Also, this tends to happen when the number of primitives/rays is ‘sufficiently’ large, as my very simple benchmark tests with only a few primitives passes without error.
If you are saying that the failure depends on the
width, height, depth argument values of your 3D launch, then an important information in your question would have been the values where it still worked and the values where it started to fail.
With the given information so far, it would be possible that you exceeded the maximum launch size limit in the newer versions.
In that case please read this post: https://forums.developer.nvidia.com/t/3d-optixlaunch-to-accommodate-multiple-viewpoints/160421/2
If you say that is related to the scene size in number of primitives, the same information about those sizes would be interesting.
Some general information about what the application does to understand the dependency between number of primitves and number of rays you mentioned would also be helpful.
An assertion inside OptiX normally indicates an issue inside OptiX itself. There might be seldom cases where this is an application error and OptiX isn’t reporting it correctly.
In that case the first thing to try is upgrading the display drivers to a newer version if available because the OptiX core implementation lives inside the driver since OptiX 6.
If that is not helping, we’d need a minimal, complete reproducer project in failing state.
Normally correct OptiX host and device code from older versions should work, except when using Selector nodes which have been removed, and there are some cases where OptiX 6.5 got pickier about some device code constructs (e.g. pairing of rtPotentialIntersection and rtReportIntersection calls) but those get the correct error messages.
Other than that, if you’re running the same old code from OptiX 3/4/5 with OptiX 6.5 (please do not use OptiX 6.0 anymore), that would mean not using the built-in triangle primitives, which will not use the hardware ray-triangle intersection on RTX boards, which is the crucial part for better performance.
I don’t think this “Attribute” error has something to do with intersection attributes. Also because your application is running in some cases, there is most likely not a systematic error with those, but note that OptiX 6 changed the intersection attribute handling due to the added built-in triangle primitives by adding attribute programs:
(In OptiX 7, intersection attributes are a set of at most 8 registers and surface attribute calculation needs to happen explicitly inside the hit programs.)