I guess there is no support for that out of the box, but I can hook in my own .dlls to access Optix?
No, that’s not how it works. Please read the Nucleus documentation first.
Generally if you want to work with data inside the Nucleus database, you’d write a connector on the client side which can retrieve the desired data from the USD files.
In case you want to do some rendering or simulation done with your own OptiX based application code, then that would effectively require to traverse an USD scene graph for the data you need and use that inside your application.
This effectively means writing a Hydra delegate. Hydra is the Pixar open-source framework to transport live scene graph data to renderers. Originally it was their OpenGL renderer in the past but got more abstract.
That in turn requires a solid understanding of the USD API first.
(In case it isn’t clear, yet, this is getting complicated.)
Even then, I have no idea how any of your application code could be made to run on server-side though. The Nucleus server machines might not even have graphics support.
Please ask your Omniverse specific questions on the sub-forums I linked above.