How to get a Cuda GpuMat into Gstreamer?

Yes, your writer pipeline uses x264enc that is CPU only. You may try this for RTP/UDP streaming.