Optimal way to read video frames in vpi using open CV and GStreamer


I am trying to read an mp4 file in Python using OpenCV CUDA and GStreamer, and use the captured frame in a VPI pipeline. Is there a way to achieve zero mem copy in the process of capturing the frame and sending it to the VPI pipeline?

Here is my code -

cap_front = cv2.VideoCapture('filesrc location=front_video.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! queue ! videoconvert ! queue ! video/x-raw, format=BGR ! appsink  ', cv2.CAP_GSTREAMER)
out = cv2.VideoWriter('appsrc ! queue ! videoconvert ! video/x-raw,format=RGBA ! nvvidconv ! nvegltransform ! nveglglessink', cv2.CAP_GSTREAMER, 0, 30.0, (1280,720))

while True:
    ret_front, frame_front = cap_front.read()

    with vpi.Backend.CUDA:
        distortion_corrected_front = vpi.asimage(frame_front)\
                                                    .remap(warpmap_distortion_correction, interp=vpi.Interp.LINEAR)\

I wanted to know if I can copy frame_front directly into a VPI buffer or other device memory to achieve zero mem copy. Also is there a way to avoid copying distortion_corrected_front into host for display.



VPI can wrap images from a preallocated buffer.
If you can read an image to a GPU-accessible buffer, you can create a VPI image with the below API:


Thanks for the reply @AastaLLL
I actually found another way of doing it using jetson_utils library. I am using the following code -

from jetson_utils import videoSource, videoOutput
import pycuda.driver as drv
import pycuda.autoprimaryctx
from pycuda.compiler import SourceModule
from extract_system_calibration import EXTRACT_SYSTEM_CALIBRATION
from exception_message import ExceptionMessage
import numpy
import cv2
import vpi

if __name__ == '__main__':

    param_videoOutput = []
    param_videoOutput.append("--width=" + str(1280))
    param_videoOutput.append(f"--height=" + str(720))

    input = videoSource('file://front_video.mp4')

    output = videoOutput('display://0', argv=param_videoOutput)

    while True:
        image = input.Capture(format='rgb8')

        if image is None:
        with vpi.Backend.CUDA:
            distortion_corrected_front = vpi.asimage(image)\
                                            .remap(warpmap_distortion_correction, interp=vpi.Interp.LINEAR)\


        if not input.IsStreaming() or not output.IsStreaming():

And the error I get is -

Traceback (most recent call last):
  File "test_script2.py", line 189, in <module>
Exception: jetson.utils -- videoOutput.Render() failed to get image pointer from first arg (should be cudaImage)
[gstreamer] gstreamer mysink taglist, video-codec=(string)"H.264\ \(Main\ Profile\)", bitrate=(uint)4686376, minimum-bitrate=(uint)3471360, maximum-bitrate=(uint)3540480;
[gstreamer] gstDecoder -- stopping pipeline, transitioning to GST_STATE_NULL
[gstreamer] gstDecoder -- onPreroll()

Is there any conversion I need to perform on the VPI image to pass it to the output streaming pipeline?



Is this topic duplicated to 249136?

Is Python essential for you?
With the C++ interface, it will be easier to get the GPU buffer pointer.


Hi @AastaLLL ,

Thank you for the reply. I was able to resolve it in Python by using jetson_utils library available at https://github.com/dusty-nv/jetson-utils. In jetson_utils, I used VideoSource and VideoOutput to capture and render video frames. I was able to pass the output of VideoSource capture, which is in cudaImage format, to the vpi pipeline and achieve zero mem copy.


This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.