I just tried out callable programs in OptiX 3.6.3 and found the same problems that were in earlier versions:
Using the Windows 64-bit dll, you cannot enable program ID exception checking. In other words, you must include:
rtContextSetExceptionEnabled( *context, RT_EXCEPTION_PROGRAM_ID_INVALID, 0 );
Otherwise, the callable program will receive junk input arguments.
Using the Windows 32-bit dll, callable programs do not work at all. The following exception is thrown if you use a callable program:
OptiX Error: Unknown error (Details: Function "_rtContextLaunch2D" caught exception: Encountered a CUDA error: result returned (700): Unknown, [6619204])
Windows 7 64-bit, CUDA 6.0, Tesla k40, driver 340.62