While I do not have experience in such an application, implementing it in CUDA will definately be easier, given the complexity of the problem. However if you could implement it in Compute Shaders, than you are vendor-independant. I would really not worry about the efficiency of interop, since roughly 98% of the compute time will be spent on calculating an image, and 2% will go to actual display. For path tracing and the likes, do not worry about interop speed. Summing it up, there are generally two things to consider:
- Is being vendor independant of value to you?
- How much learning time are you willing to invest in either technology that you are missing right now to go about implementing a path tracer?
I would say, if you are willing to take the time and learn Compute Shaders (OpenGL 4.3 is getting friendlier in terms of types and capability, however the OpenGL statemachine is still a pain), then you should go with OpenGL. Being able to run on “the evil” HW might prove to give significantly better performance, since GCN architecture favors this kind of workload. Qt 5.1 brings about very good helper classes that facilitate OpenGL coding. Coding it in CUDA will most likely be easier and faster, so it pretty much boils down to what your preferences are.