Problem using alloca with OptiX

That might be on purpose. I’d need to check with the core team.
Yeah, the pipeline stack size calculation wouldn’t be able to figure that out today.

There is no workaround for “unimplemented PTX intrinsics” errors like that. These can only be solved in future driver releases when possible.

Standard cudaAlloc and cudaFree calls are not supposed to work inside OptiX device code and would also be super slow. There were times when they worked inside OptiX 7 versions but I think these are actively forbidden now.

For dynamic allocations inside device code you would usually allocate some buffer upfront and then grab chunks from that using atomics building some list. For varying output result sizes that is described here:
https://forums.developer.nvidia.com/t/whats-your-solution-to-get-all-hit-primitives-of-multiple-rays/239528/2
Though freeing those dynamically would require some logic. You’d need something like an arena allocator maintaining allocated and free blocks. I haven’t implemented that in device code.