The standard way of initializing your ray payload is to do it before calling optixTrace(). For this case, where you’re asking about shadow rays, you can put the initialization wherever the shadow ray is cast, it might be in a closest-hit shader, or in raygen. For primary rays, payload initialization would always go in the raygen program immediately prior to calling optixTrace().
Any-hit will be called before closest-hit, so your code might work for you in that sense. But, don’t forget that any-hit shaders are not expected to be called in depth order, so if you are initializing something in your payload for use in any-hit shaders, it won’t work. Also calling optixTerminateRay() can work for opaque shadows, but if you need to accumulate your attenuation with transparent objects, you will need something more sophisticated in order to sort your any-hit intersections in depth order and composite them properly.
Bigger picture here is that if you are casting shadow rays against RTX enabled triangle meshes, you should consider disabling your any-hit shader completely (OPTIX_GEOMETRY_FLAG_DISABLE_ANYHIT or OPTIX_INSTANCE_FLAG_DISABLE_ANYHIT or OPTIX_RAY_FLAG_DISABLE_ANYHIT), and instead use closest-hit shaders and the the flag OPTIX_RAY_FLAG_TERMINATE_ON_FIRST_HIT.
It used to be the case in OptiX 5 and earlier, before there was hardware acceleration, that using any-hit programs for shadows and terminating the ray was the most performant way to do shadowing. With the hardware acceleration, now the recommended approach is to avoid using any-hit programs completely, if you can.