Optimal way to read video frames in vpi using open CV and GStreamer

Hi,

I am trying to read an mp4 file in Python using OpenCV CUDA and GStreamer, and use the captured frame in a VPI pipeline. Is there a way to achieve zero mem copy in the process of capturing the frame and sending it to the VPI pipeline?

Here is my code -

cap_front = cv2.VideoCapture('filesrc location=front_video.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! queue ! videoconvert ! queue ! video/x-raw, format=BGR ! appsink  ', cv2.CAP_GSTREAMER)
out = cv2.VideoWriter('appsrc ! queue ! videoconvert ! video/x-raw,format=RGBA ! nvvidconv ! nvegltransform ! nveglglessink', cv2.CAP_GSTREAMER, 0, 30.0, (1280,720))

while True:
    ret_front, frame_front = cap_front.read()

    with vpi.Backend.CUDA:
        distortion_corrected_front = vpi.asimage(frame_front)\
                                                    .convert(vpi.Format.NV12_ER)\
                                                    .remap(warpmap_distortion_correction, interp=vpi.Interp.LINEAR)\
                                                    .convert(vpi.Format.RGB8)
        out.write(distortion_corrected_front.cpu())

I wanted to know if I can copy frame_front directly into a VPI buffer or other device memory to achieve zero mem copy. Also is there a way to avoid copying distortion_corrected_front into host for display.

Thanks

Hi,

VPI can wrap images from a preallocated buffer.
If you can read an image to a GPU-accessible buffer, you can create a VPI image with the below API:
https://docs.nvidia.com/vpi/group__VPI__Image.html#ga3e7cf2520dd568a7e7a9a6876ea7995c

Thanks.

Thanks for the reply @AastaLLL
I actually found another way of doing it using jetson_utils library. I am using the following code -

from jetson_utils import videoSource, videoOutput
import pycuda.driver as drv
import pycuda.autoprimaryctx
from pycuda.compiler import SourceModule
from extract_system_calibration import EXTRACT_SYSTEM_CALIBRATION
from exception_message import ExceptionMessage
import numpy
import cv2
import vpi

if __name__ == '__main__':

    param_videoOutput = []
    param_videoOutput.append("--width=" + str(1280))
    param_videoOutput.append(f"--height=" + str(720))

    input = videoSource('file://front_video.mp4')

    output = videoOutput('display://0', argv=param_videoOutput)

    while True:
        image = input.Capture(format='rgb8')

        if image is None:
            continue
        
        with vpi.Backend.CUDA:
            distortion_corrected_front = vpi.asimage(image)\
                                            .convert(vpi.Format.NV12_ER)\
                                            .remap(warpmap_distortion_correction, interp=vpi.Interp.LINEAR)\
                                            .convert(vpi.Format.RGB8)      

        output.Render(distortion_corrected_front)

        if not input.IsStreaming() or not output.IsStreaming():
            break

And the error I get is -

Traceback (most recent call last):
  File "test_script2.py", line 189, in <module>
    output.Render(distortion_corrected_front)
Exception: jetson.utils -- videoOutput.Render() failed to get image pointer from first arg (should be cudaImage)
[gstreamer] gstreamer mysink taglist, video-codec=(string)"H.264\ \(Main\ Profile\)", bitrate=(uint)4686376, minimum-bitrate=(uint)3471360, maximum-bitrate=(uint)3540480;
[gstreamer] gstDecoder -- stopping pipeline, transitioning to GST_STATE_NULL
[gstreamer] gstDecoder -- onPreroll()

Is there any conversion I need to perform on the VPI image to pass it to the output streaming pipeline?

Thanks

Hi,

Is this topic duplicated to 249136?

Is Python essential for you?
With the C++ interface, it will be easier to get the GPU buffer pointer.

Thanks.

Hi @AastaLLL ,

Thank you for the reply. I was able to resolve it in Python by using jetson_utils library available at https://github.com/dusty-nv/jetson-utils. In jetson_utils, I used VideoSource and VideoOutput to capture and render video frames. I was able to pass the output of VideoSource capture, which is in cudaImage format, to the vpi pipeline and achieve zero mem copy.

Suraj

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.