In Optix, How much memory can a single optixLaunch allocate?

What is your system configuration?
OS version, installed GPU(s), VRAM amount, display driver version, OptiX major.minor.micro version, CUDA toolkit version used to generate the module inputs, host compiler version.

The answer to your topic’s title is: As much VRAM as is installed on your GPU. It depends on what you programmed.

The amount of memory really allocated for the OptiX kernel depends a lot on the required local memory and the OptiX stack size, which in turn depends on the ray tracing algorithms (traversal depth, max recursions, and how much memory the program domains use).

In your case your per ray payload structure is rather big with1460 bytes if I counted that correctly.
You define that inside the ray generation program. That means it’s allocated per per thread, means launch dimension size.
Then I assume you pass a pointer to that payload structure in two 32bit payload registers in optixTrace.

You should always calculate the OptiX stack size and look at the resulting parameters. Mind that there is a hard upper limit for the stack size which is 64kB.
Since that is allocated per thread as well, there can easily be some GBs used just by that.
Example code inside the OptiX SDK examples or here:

The general recommendation is to reduce the required stack size as much as possible for performance and memory reasons.
Use iterative instead of recursive algorithms.
Don’t define too much local data inside the individual program domains. (If you generate PTX code for the modules and not OptiX-IR, then you can look at the “local depot” sizes at the beginning of the programs.

That you get an an illegal memory access error instead of an out of memory error means that you accessed something out of bounds.
When changing the array sizes inside that Payload structure, have you made sure all other source code also accesses these arrays inside their bounds? (I would define that size 50 with a single define instead to make sure all code adheres to the sizes.)

All your elements inside the Payload structure have a CUDA memory alignment requirement of 4 bytes, so there shouldn’t be inadvertent padding inside that struct or misaligned memory accesses which would have reported such different error message.

Related threads to figure out how much memory is used overall:

If you haven’t enabled OptiX validation mode when debugging this, here is example code doing that.
Check if there is additional information with that enabled.

(PS: To all readers: Please do not attach screenshots of source code or command prompts on developer forums.
Instead just copy and paste the text itself into a code block for better readability and easier handling.)