rtContextValidate fails after rtContextSetRayGenerationProgram unless it was also called before

My small ray tracing application has a very weird behaviour / bug. Calling rtContextValidate after setting up everything, either directly or indirectly via rtContextLaunch, results in a RT_ERROR_TYPE_MISMATCH.

However, if I add a call to rtContextValidate at any location in the code before calling rtContextSetRayGenerationProgram, the error goes away and the code produces the expected result!

I have no idea what could be causing this. According to what I have read in the docs so far, a call to rtContextValidate that did not encounter any problems should never change the outcome of a subsequent call to rtContextValidate, right?

Right now, I have only two potential explanations for this: either I did something nasty with buffers or something that causes undefinded behaviour, or there is a bug in OptiX. Unfortunately, I have no idea how to find out what exactly is going on here.

Btw: I am using OptiX 4.0.1 with CUDA 8.0 on Windows 10 (VS 2015)

The first thing you should try is updating to OptiX 4.0.2 and see if the problem persists.

If yes, I would go through all cases inside the OptiX API Reference document and check which calls can trigger the RT_ERROR_TYPE_MISMATCH and see if the code around that is correctly specifying the necessary objects at the proper size and indices.

Check with rtContextGetErrorString if there are more information available about the type mismatch error.

Maybe add individual validation calls for other objects than the whole context to see if you can pinpoint this problem.

Additionally I would enable all exceptions and add an exception program which prints the exception code during debugging to see if the result is really generated correctly or if there are any other potential problems. I’ve posted code doing this on this forum before. You’ll find it with the search option in the top right on this site.
Note that compilation and runtime performance is affected in OptiX 4.0.2 when enabling exceptions. Only use that for debugging, never during benchmarks.

What GPU(s) do you use?

Thanks for the quick reply and helpful tips. I found the issue, which was pretty simple and stupid. I accidentally assigned a uint to an int variable of my intersection program.

Apparently, validate only detects this after the ray generation programms have been added. I guess there is some kind of “already validated” flag or something that prevents it from finding this issue if a validate was called previously.

The behaviour persists in 4.0.2

Declaring a variable like this in my intersection program:

rtDeclareVariable(int, mesh_type, , );

and assigning it like this

optix::Geometry geom = ctx_->createGeometry();
// ...
geom["mesh_type"]->setUint(type);

Causes the issue.

I am using a single GTX 970 btw

As far as why OptiX only detects the error when a ray-gen program is attached:

The “validate” command looks at the entire context and tries to find everything that is reachable during rendering, then validates that set of things. If a piece of geometry is not used – nothing traces rays against its GeometryGroup, for example – then the intersection program doesn’t necessarily have to be checked. OptiX may ignore it entirely.

Does that explain the behavior you’re seeing? Is there any other way to trigger the intersection program other than by shooting a ray from the ray-gen program?