You’re not actually handling the cutout opacity in your any hit program.
If you want to support materials with cutout opacity, then both the radiance and the shadow ray for those material need to have an anyhit program attached.
In both anyhit programs you need to calculate the current opacity and if it’s a cutout (a hole in the surface, usually for opacity < 1.0f) than you must call optixIgnoreIntersection() to let the ray continue through the surface.
If it’s opaque, you need to do nothing in the anyhit program on the radiance ray to let the traverse keep searching for the closest hit.
But inside the shadow ray anyhit program you should call optixTerminateRay() to let the opaque surface parts block the visibilty. No need to keep traversing the BVH when the visibility is false from any surface in the shadow ray t_min, t_max interval.
Please read this thread which explains exactly the same things and also touches on light attenuation pitfalls in Whitted renderers:
https://forums.developer.nvidia.com/t/anyhit-program-as-shadow-ray-with-optix-7-2/181312
(This forum has a search field in the top right!)
Please follow the links to my OptiX 7 example code in there as well. They contain everything to support stochastic cutout opacity in a progressive path tracer.