I am using RT cores to accelerate general computing. Does Optix support dynamic parallelism? I want to use optixTrace() in dynamic parallelism.
No, you cannot launch other kernels inside OptiX device code and the only way to launch OptiX kernels is with the optixLaunch host call, not the CUDA runtime chevrons used for dynamic parallelism.
Please read these OptiX Programming Guide chapters:
https://raytracing-docs.nvidia.com/optix7/guide/index.html#introduction#overview
https://raytracing-docs.nvidia.com/optix7/guide/index.html#program_pipeline_creation#programming-model