Please have a look into this thread and follow the links in there: https://forums.developer.nvidia.com/t/optix-7-0-0-memory-error/180777
In theory you can have a as many optixTrace() call invocations inside your ray tracing kernel as you want when staying under the maximum launch dimension.
But you will compete against the operating system for the GPU resources.
If you are running on a GPU dedicated to compute tasks only (Tesla Compute Cluster (TCC) driver mode), there is no OS timeout like under the Windows Display Driver Model (WDDM since Vista, it got better in Windows 10 with WDDM2).
The RT cores on the top end Turing GPUs could already handle over 11 GRays/second. With complex scenes or shaders you will usually run into memory bandwidth limits first.
It’s usually the simplest approach to partition work into reasonably sized chunks and launch less work more often instead of using some super expensive kernel which does it all.
There are plenty of options in computer graphics to split work, e.g. into individual samples in a progressive Monte Carlo algorithm, into tiles of an image, into an image per light source etc.