Not really.
-
The optixTrace() call arguments are incorrect.
You used the RAY_TYPE_SHADOW for the flags argument and hardcoded the SBToffset, SBTstride, and missSBTIndex to zero.
If RAY_TYPE_SHADOW is not zero, then this is not going to reach your anyhit and closesthit shaders for the shadow ray at all. -
If your shadow attenuation is only a single float, then it’s faster to implement that as a single payload register as in the code I posted above.
In my examples that is using the per ray payload because that code is a stripped down versions of an MDL-capable renderer which could evaluate procedural coutout opacity materials inside the anyhit program and that evaluation needed a lot more state than a single boolean.
You would only need to change the single payload register to be interpreted as float by using the necessary __uint_to_float() and __float_to_uint() device reinterpret cast functions.
Otherwise the code structure is OK, but that anyhit program would only result in fully opaque materials so far.
Again, the link to my example code above contains the necessary calls and programs which show exactly the structure you just described. Here’s my shadow ray optixTrace call.
You would need to use the structure of the __anyhit__shadow_cutout() example and change the Monte Carlo binary decision for cutout or not to your desired attenuation float value, and how that is done for some glass material is shown inside the __anyhit__glass_occlusion() program inside the OptiX SDK optixWhitted example.
I forgot to mention an important detail related to the BVH structure which will break this attenuation or any counting method via anyhit programs: Since the BVH builds are splitting primitives per default for performance reasons, anyhit programs can be invoked from the same primitive multiple times. In this case that will result in broken shadows.
To prevent that, you must set the OPTIX_GEOMETRY_FLAG_REQUIRE_SINGLE_ANYHIT_CALL.
Please read the OptiX Programming Guide chapter on Acceleration Structures for the different build flags, explained below Listing 5.5 which says exactly that.
Related link: https://forums.developer.nvidia.com/t/mesh-artifacts-when-using-anyhit-for-transparency-optix-7/156600/2