Hi Rémy,
it seems that you want to do something similar to what I am already doing. You can try our code, which is public and with MIT license. It supports reflections and depolarization, and simple diffraction is in progress:
If you find it useful, please cite our papers.
Kind regards,