Optimal way to read video frames in vpi using open CV and GStreamer

Robotics & Edge Computing Jetson Systems Jetson Xavier NX
, , , , ,
1

Hi,

I am trying to read an mp4 file in Python using OpenCV CUDA and GStreamer, and use the captured frame in a VPI pipeline. Is there a way to achieve zero mem copy in the process of capturing the frame and sending it to the VPI pipeline?

Here is my code -

cap_front = cv2.VideoCapture('filesrc location=front_video.mp4 ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! queue ! videoconvert ! queue ! video/x-raw, format=BGR ! appsink  ', cv2.CAP_GSTREAMER)
out = cv2.VideoWriter('appsrc ! queue ! videoconvert ! video/x-raw,format=RGBA ! nvvidconv ! nvegltransform ! nveglglessink', cv2.CAP_GSTREAMER, 0, 30.0, (1280,720))

while True:
    ret_front, frame_front = cap_front.read()

    with vpi.Backend.CUDA:
        distortion_corrected_front = vpi.asimage(frame_front)\
                                                    .convert(vpi.Format.NV12_ER)\
                                                    .remap(warpmap_distortion_correction, interp=vpi.Interp.LINEAR)\
                                                    .convert(vpi.Format.RGB8)
        out.write(distortion_corrected_front.cpu())

I wanted to know if I can copy frame_front directly into a VPI buffer or other device memory to achieve zero mem copy. Also is there a way to avoid copying distortion_corrected_front into host for display.

Thanks

3

Hi,

VPI can wrap images from a preallocated buffer.
If you can read an image to a GPU-accessible buffer, you can create a VPI image with the below API:
https://docs.nvidia.com/vpi/group__VPI__Image.html#ga3e7cf2520dd568a7e7a9a6876ea7995c

Thanks.

4

Thanks for the reply @AastaLLL
I actually found another way of doing it using jetson_utils library. I am using the following code -

from jetson_utils import videoSource, videoOutput
import pycuda.driver as drv
import pycuda.autoprimaryctx
from pycuda.compiler import SourceModule
from extract_system_calibration import EXTRACT_SYSTEM_CALIBRATION
from exception_message import ExceptionMessage
import numpy
import cv2
import vpi

if __name__ == '__main__':

    param_videoOutput = []
    param_videoOutput.append("--width=" + str(1280))
    param_videoOutput.append(f"--height=" + str(720))

    input = videoSource('file://front_video.mp4')

    output = videoOutput('display://0', argv=param_videoOutput)

    while True:
        image = input.Capture(format='rgb8')

        if image is None:
            continue
        
        with vpi.Backend.CUDA:
            distortion_corrected_front = vpi.asimage(image)\
                                            .convert(vpi.Format.NV12_ER)\
                                            .remap(warpmap_distortion_correction, interp=vpi.Interp.LINEAR)\
                                            .convert(vpi.Format.RGB8)      

        output.Render(distortion_corrected_front)

        if not input.IsStreaming() or not output.IsStreaming():
            break

And the error I get is -

Traceback (most recent call last):
  File "test_script2.py", line 189, in <module>
    output.Render(distortion_corrected_front)
Exception: jetson.utils -- videoOutput.Render() failed to get image pointer from first arg (should be cudaImage)
[gstreamer] gstreamer mysink taglist, video-codec=(string)"H.264\ \(Main\ Profile\)", bitrate=(uint)4686376, minimum-bitrate=(uint)3471360, maximum-bitrate=(uint)3540480;
[gstreamer] gstDecoder -- stopping pipeline, transitioning to GST_STATE_NULL
[gstreamer] gstDecoder -- onPreroll()

Is there any conversion I need to perform on the VPI image to pass it to the output streaming pipeline?

Thanks

5

Hi,

Is this topic duplicated to 249136?

Is Python essential for you?
With the C++ interface, it will be easier to get the GPU buffer pointer.

Thanks.

7

Hi @AastaLLL ,

Thank you for the reply. I was able to resolve it in Python by using jetson_utils library available at https://github.com/dusty-nv/jetson-utils. In jetson_utils, I used VideoSource and VideoOutput to capture and render video frames. I was able to pass the output of VideoSource capture, which is in cudaImage format, to the vpi pipeline and achieve zero mem copy.

Suraj