How to pass data between GStreamer and CUDA without memory copying?

Hi folks.

I’m trying to stitch multiple IP camera’s video streams into a single video by GStreamer and CUDA.

There are 3 blocks. The first block receives and decodes video streams by GStreamer. The second block stitches the decoded video frames by CUDA. And last block display the stitched video frame by GStreamer.

My pipeline is as follows.

( rtspsrc ! decodebin ! nvvidconv ! appsink ) --+--> ( CUDA process ) ---> ( appsrc ! nvvidconv ! nvoverlaysink )
                     :                          |
  (same number of pipelines as cameras)         |
                     :                          |
( rtspsrc ! decodebin ! nvvidconv ! appsink ) --+

This pipeline works. However,it has a performance problem about data passings between GStreamer and CUDA because I copy the data to memory space that can be handled by each.

I’d like to avoid the copying, and I guess NVMM makes it unnecessary. But I don’t know how to do it.
Do you have any ideas?

You may have a look to gstreamer nvivafilter plugin. Check https://devtalk.nvidia.com/default/topic/938761/jetson-tx1/how-to-use-the-gstreamer-element-nvivafilter-/post/4986671/#4986671.

Thank you for the response.

However, nvivafilter can process only one input stream. I have to process multiple streams to stitch them.

Or is there a good way with nvivafilter to do it?

Hi KentaroH,
gstreamer cannot run your case without memcpy(). We suggest try tegra_multimedia_api.

Hi DaneLLL,

OK.
Actually, I was planning to use tegra_multimedia_api as another way not GStreamer.
It’s about time.

Thank you.

Hi KentaroH,

You may found interesting the following information about the GstCUDA framework, I think that is exactly what you are looking for.

GstCUDA is a RidgeRun developed GStreamer plug-in enabling easy CUDA algorithm integration into GStreamer pipelines. GstCUDA offers a framework that allows users to develop custom GStreamer elements that execute any CUDA algorithm. The GstCUDA framework is a series of base classes abstracting the complexity of both CUDA and GStreamer. With GstCUDA, developers avoid writing elements from scratch, allowing the developer to focus on the algorithm logic, thus accelerating time to market.

GstCUDA offers a GStreamer plugin that contains a set of elements, that are ideal for GStreamer/CUDA quick prototyping. Those elements consist in a set of filters with different input/output pads combinations, that are run-time loadable with an external custom CUDA library that contains the algorithm to be executed on the GPU on each video frame that passes through the pipeline. GstCUDA plugin allows users to develop their own CUDA processing library, pass the library into the GstCUDA filter element that best adapts to the algorithm requirements, executes the library on the GPU, passing upstream frames from the GStreamer pipeline to the GPU and passing the modified frames downstream to the next element in the GStreamer pipeline. Those elements were created with the CUDA algorithm developer in mind - supporting quick prototyping and abstracting all GStreamer concepts. The elements are fully adaptable to different project needs, making GstCUDA a powerful tool that is essential for CUDA/GStreamer project development.

One remarkable feature of GstCUDA is that it provides a zero memory copy interface between CUDA and GStreamer on Jetson TX1/TX2 platforms. This enables heavy algorithms and large amounts of data (up to 2x 4K 60fps streams) to be processed on CUDA without the performance caused by copies or memory conversions. GstCUDA provides the necessary APIs to directly handle NVMM buffers to achieve the best possible performance on Jetson TX1/TX2 platforms. It provides a series of base classes and utilities that abstract the complexity of handle memory interface between GStreamer and CUDA, so the developer can focus on what actually gives value to the end product. GstCuda ensures an optimal performance for GStreamer/CUDA applications on Jetson platforms.

You can find detailed information about GstCUDA on the following link:
http://developer.ridgerun.com/wiki/index.php?title=GstCUDA

I hope this information can be useful to you.

Best regards,
-Daniel