Electromagnetic wave simulation using OptiX

Depends on your reflection distribution function and your transmitter and receiver sizes.

If your reflections are specular (== the bi-directional reflection distribution function (BRDF) is a Dirac function), then the probability to hit an infinitely small point (== a specific reflection vector) when shooting rays randomly is zero!
That’s like a singular light which needs to be sampled explicitly for direct lighting, but that in turn is not done for purely specular surfaces because, again there exists only one path connecting along a specular reflection.

If the BRDF is not specular, then it’s possible to connect paths from receiver to surface to transmitter the way I described in the algorithm inside the links above. That is like a direct lighting (next event estimation) algorithm.

If your reflections are really specular, a totally different approach than a unidirectional Monte Carlo path tracer would need to be taken.

Before we are using Optix, we are doing this with image receiver, which we find the image with respect to the surface and link the image receiver with transmitter and find the point.

Yes, that sounds like you’re analyzing surface points to find the ones which build a pair of input, output directions which fulfill the specular reflection condition.
That’s basically like mutating a surface hit position until it fulfills the specular reflection condition. If that is not inside the bounds of the surface area, there is no path connecting in one bounce from the current input direction to the receiver (or transmitter depending on where you started your search).
Means if you convert a flat surface into a plane equation and then calculate the point where the input and output directions are exactly a specular reflection, and then check if that point is inside the limited area of the original surface (triangle), that could connect a transmitter with a receiver with one specular bounce.
If there are multiple specular bounces the whole path would need to be mutated.
That would all not require ray tracing, unless the resulting path would need to be checked for visibility against objects inside the scene.

But for OptiX, we always traverse in the whole scenario, so if this surface is part of a cube, it cannot successfully find the reflection point still.

I don’t know what your simulation scenario consists of or what you mean with “if this surface is part of a cube”.
For additional ideas you would need to add a lot more information about what your simulation scenario is exactly (scene contents, BRDF and receiver and transmitter properties).

If your simulation runs inside a specular reflecting cube, there is a totally different and very simple method to connect a transmitter with a receiver if they are infinitely small points which doesn’t need raytracing at all. I described this here: Reflection in Optix Prime - #2 by droettger