Hello,
I would like to ask, if there is any way to create my own custom accelerating structure that uses RT cores?
Thank you.
Nope.
Acceleration structures and their traversal are abstracted by ray tracing APIs (OptiX, DXR, Vulkan RT) because their implementation is free to change among software and GPU hardware product releases.
For example different OptiX versions had various different acceleration structure and traversal implementations in the past (e.g. there was even a kD-tree version) and this is also changing among GPUs, which is one reason why there are RT core version and acceleration structure compatibility query functions in OptiX 7.