For a start, please have a look at this thread:
https://forums.developer.nvidia.com/t/some-questions-about-ray/250279/2
That explains when raygeneration, intersection, anyhit, closesthit and miss programs are called and what can influence that with geometry flags during AS build, ray flags in optixTrace, and how anyhit programs can influence the traversal with optixTerminateRay and optixIgnoreIntersection.
It’s basically this:
The raygeneration program is the entry point of your pipeline and it can call optixTrace.
(Only some program domains are allowed to call optixTrace: raygen, closesthit, miss.)
Scene BVH traversal starts at the given traversable handle of the optixTrace call.
If there is any ray-AABB intersection, the intersection program is called. (The intersection program is implicit for built-in triangles.)
If there is a potential intersection reported, it calls into the optional anyhit program.
If the anyhit program calls optixIgnoreIntersection the ray.tmax value stays the same and scene traversal continues,
If the anyhit program calls optixTerminateRay, scene traversal stops and the closesthit program is called with that ray.tmax.
If there is no anyhit program, scene traversal continues until the closest intersection distance is found and set in ray.tmax and the closest hit program is called.
(In either anyhit or closesthit programs, the hit attribute registers of the reported intersection (e.g. two for barycentric coordinates on a triangle) provide the information allowing the calculation of the per-hit vertex attributes of your geometric primitive.)
If there is no closesthit program, nothing happens at that point.
If nothing is hit (or the anyhit program ignored all intersections) then the miss program is called.
If either the closesthit or miss program were reached (if provided or not), the ray traversal ends and returns to the code after the last called optixTrace.
If anything goes wrong with the optixTrace (illegal ray values, stack overflow, etc.), the optional or a default exception program is called. Control does not return to the location that triggered the exception, and execution of the launch index ends.
Callable programs are basically function tables in your pipeline which can be called by an index, which allows a flexible configuration of programs. Continuation callables can call optixTrace.
This table of device functions shows which functions may be called in what program domain.
If you check that table for the callables, that is probably what the above image wanted to express. Because continuation callable programs can call optixTrace, they are only allowed in raygen (where they don’t make much sense), closesthit and miss programs, while direct callable programs can be called in all program domains other than the exception program. That’s shown with the arrows in the above image.