Shadow Ray from Closest Hit

Hello all,

Please forgive if this is a simple question, but I would like to create a shadow ray with OptiX 7.2 such that it is called from the closest hit shader.

For example, I have some value (say, solar illumination) that will determine if the shadow ray is traced. I would also like the result to show an image beneath a plane that may have hole(s) in it - multiple calls to closest hit maybe (?, I am not sure).

A cut-out that will show geometry below (if it exists) as well as the shadow(s).

Can this be done?

Snippet of pseudo-code:

extern "C" __global__ void __closesthit__ch( ) { 
    // calculate solar illumination here
    float solar = dot(normalDir, sunDir);
    if(solar > 0.0f) {
        // Call trace with "shadow" ray(s) somehow

Thanks for any assistance, maybe I’m a little slow but this seems to have me stumped :)

Hi @picard1969,

Yes you can trace a shadow ray from your closest-hit program. To do this, you call optixTrace() inside your if(solar) block, and you will pass your shadow ray type to optixTrace. Be aware that this counts as a recursive trace call, and so you will need to adjust your maximum trace depth and stack size appropriately. You will also need to understand ray types and adjust your SBT to allow for two types: primary rays and shadow rays. I would recommend opening the OptiX Programming Guide and searching for a few of these terms using the search box feature at the top right. NVIDIA OptiX 7.3 - Programming Guide For example, search for “ray type”, and search for “recur”, and click on and read through all the results you get.

Whether your shadow ray needs to use closest hit is up to you. Often you don’t need closest hit calls, because all you need to know is whether an occlusion occurred. You can do that using only a miss program for the shadow ray type.

Also, you don’t necessarily need to trace your shadow ray in your primary ray type’s closest hit program. You could pass the primary ray’s hit results back to raygen via the payload, and then trace your shadow ray in raygen. This is optional, either way is okay. calling optixTrace() from your primary ray’s closest hit program is easier, like in your pseudocode. The advantage of moving it to raygen is then you can eliminate the recursive call, and you can reduce your stack size and maximum trace depth. Sometimes doing this “iterative” formulation is also more performant than the recursive formulation.

For cutouts, you can study the SDK sample called optixCutouts. This is an example of using an any-hit program to dynamically cut holes in a surface. You can evaluate holes any way you like, you simply call optixIgnoreIntersection() if you want there to be a hole in the surface, and that will cause OptiX to not treat it as a hit, not call your closest hit program, and continue tracing looking for another hit.


Thanks @dhart for the response and information. Some great stuff contained, however I still have some issues. I have been racking my tiny brain for a couple of days on this one and I must be overlooking something obvious.

Currently I have placed the shadow ray when __anyhit__ is called. It looks like it properly renders a cutout in an image but nothing underneath when there is valid geometry beneath, which should not be happening. I suspect this could be happening because I am not allowing the ray to recurse as it should be (just a guess).

The __closesthit__ shader snippet pseudo code follows:

extern "C" __global__ void __closesthit__ch( ) {
    // Calculate solar illumination here
    float solar = dot(normalDir, sunDir);
    if(solar > 0.0f) {
        RayDataShade shd;
        // Store attenuation in shd
        shd.attenuation = 1.0f;
       uint2 pay = splitShade(shd);   // acts like example split and merge functions from OptiX 7.2

        optixTrace(topObject, hitPt, sunDir, fmaxf(epsilon,0.0f), DEFAULT_MAX, 0.0f, 
                           OptixVisibilityMask(0xFF), OPTIX_RAY_FLAG_DISABLE_CLOSESTHIT,
                           RAYTYPE_SHADOW, NUM_RAYTYPES, RAYTYPE_SHADOW, pay.x, pay.y);

        if(shd.attenuation > 0.0f) {
            // Render some stuff 

The associated __anyhit__ shader pseudo code follows:

extern "C" __global__ void __anyhit__shd( ) {
    RayDataShade *shd = mergeShade(optixGetPayload_0( ), optixGetPayload_1( ));
    if(shd->attenuation == 1.0f) {
        shd->attenuation = 0.0f;

I apologize if this is something stupid I am not seeing, but it has been a long day :) - any help would be great.

You’re not actually handling the cutout opacity in your any hit program.

If you want to support materials with cutout opacity, then both the radiance and the shadow ray for those material need to have an anyhit program attached.
In both anyhit programs you need to calculate the current opacity and if it’s a cutout (a hole in the surface, usually for opacity < 1.0f) than you must call optixIgnoreIntersection() to let the ray continue through the surface.

If it’s opaque, you need to do nothing in the anyhit program on the radiance ray to let the traverse keep searching for the closest hit.
But inside the shadow ray anyhit program you should call optixTerminateRay() to let the opaque surface parts block the visibilty. No need to keep traversing the BVH when the visibility is false from any surface in the shadow ray t_min, t_max interval.

Please read this thread which explains exactly the same things and also touches on light attenuation pitfalls in Whitted renderers:
(This forum has a search field in the top right!)

Please follow the links to my OptiX 7 example code in there as well. They contain everything to support stochastic cutout opacity in a progressive path tracer.

Thanks @droettger for information and link.

I will better use the search option next time.