There is basically only one case where anyhit programs need to be used when doing visibility (shadow) rays in OptiX 7 and that is when the surface can have cutout opacity, means holes inside the surface of thin-walled geometry which need to be determined per ray if they are texture based or procedural.
In that case your shadow ray hit record only needs the anyhit program and that calls optixIgnoreIntersection when it determines the ray went through a hole inside the hit primitive and optixTerminateRay when not.
You can find examples for that inside the OptiX SDK example optixCutouts or in my OptiX 7 examples:
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/rtigo3/shaders/anyhit.cu#L94
This will slow down the hardware BVH traversal on RTX boards because it’s calling back into the streaming multiprocessor to handle the user defined anyhit program.
Now, if your scene never uses cutout opacity, a visibility ray can be implemented with just a miss program!
Means no callback into closesthit or anyhit programs is required at all because all you want to know is if anything is inside the ray’s [t_min, t_max] interval.
This should be the fastest method in OptiX 7 to implement visibility rays, when there is no cutout opacity inside the scene.
For that you shoot rays with the flags
OPTIX_RAY_FLAG_DISABLE_ANYHIT | OPTIX_RAY_FLAG_DISABLE_CLOSESTHIT | OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT
That means the ray will neither call into closesthit, nor anyhit programs and will stop traversal on the very first hit, which isn’t necessarily the closest hit but rather, like the usual anyhit behavior, the first hit which is inside the [t_min, t_max] interval, means something is blocking the visibility.
For built-in triangles not even a hit record program group would be required for that ray type, so you wouldn’t even need SBT hit records for such visibility ray types at all, only its miss record.
Or if that is for custom primitives with an intersection program, the program group for that hit record has nullptr for closesthit and anyhit entries and the usual SBT layout with hit record entries per ray type.
// The shadow ray is only a single payload to indicate the visibility test result.
// Default to visibilty being blocked by geometry. If the miss shader is reached this gets set to 1.
unsigned int isVisible = 0;
// Note that the sysData.sceneEpsilon is applied on both sides of the shadow ray [t_min, t_max] interval
// to prevent self-intersections with the actual light geometry in the scene.
optixTrace(sysData.topObject,
thePrd->pos, lightSample.direction, // origin, direction
sysData.sceneEpsilon, lightSample.distance - sysData.sceneEpsilon, 0.0f, // tmin, tmax, time
OptixVisibilityMask(0xFF),
OPTIX_RAY_FLAG_DISABLE_ANYHIT | OPTIX_RAY_FLAG_DISABLE_CLOSESTHIT | OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT,
0, 0, TYPE_RAY_SHADOW, // The shadow ray type only uses the miss program.
isVisible);
if (!isVisible)
{
return;
}
The visibility ray’s miss shader just need to set the single payload register to flag that nothing was hit in the interval.
To say the obvious, that means if the ray reaches the miss shader nothing blocked the visibility.
extern "C" __global__ void __miss__shadow()
{
optixSetPayload_0(1); // isVisible != 0
}
The OptixProgramGroup setup then look like this for such a visibility ray:
...
pgd = &programGroupDescriptions[PGID_MISS_SHADOW];
pgd->kind = OPTIX_PROGRAM_GROUP_KIND_MISS;
pgd->flags = OPTIX_PROGRAM_GROUP_FLAGS_NONE;
pgd->miss.module = moduleMiss;
pgd->miss.entryFunctionName = "__miss__shadow"; // The shadow ray only needs this miss shader if all materials are opaque. No hit shaders for shadow rays!
...
If you actually need to implement “shadow rays” which adjust the attenuation when passing through transparent objects then you would need to enhance the anyhit program to change your attenuation depending on the “amount of opacity”.
That’s incorrect inside an unbiased global illumination light transport algorithm. There visibility is a binary decision and caustics are handled by the light transport.
It’s normally used in Whitted-like renderers to enhance shadows, and in some cases for optimization purposes if direct lighting should pass through thin-glass geometry, resp. when not considering refractions.
Doing something like cutout opacity or shadow attenuation inside the closesthit program will not be cheaper because reaching the closesthit program means the ray has ended and you’d need to shoot continuation rays to capture the remaining effects.