Hi,
You can implement the function in CUDA code. With NvBufSurface APIs, you can access and modify data of each pixel. There is a CUDA sample for gstreamer nvivafilter plugin:
Nvcompositor plugs alpha is not work and alpha plugs no work - #14 by DaneLLL
The sample demonstrates applying alpha effect to each pixel. Not exactly identical to your use-case, but can be a reference.