I had to ask James for this one :-). His response:
Does OptiX in any way use shared memory to optimize access to buffers?
No. OptiX current uses shared memory only for certain intra-warp communication and SM wide load balancing. Note that you should not use shared memory directly in your own programs, because OptiX manages all the shared memory and may not leave any available for your programs.
How does OptiX manage the stack, ie. if i have some big struct in the payload (random number state), is it passed up and down?
Stack memory is currently stored in local memory. At this time ray payloads are passed by pointer and not by value, so there is no copying of the ray payload occurring when passing the payload around. You will of course get copying if you do rpl = rpl2.
Note that these are current implementation details and can of course change for future versions of OptiX.