Extract frame from nvvm memory to gpu (memory:NVMM, appsink)

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
1

Hi everyone

I have build a gst-pipeline with appsink (emit-signals=True).
Device: Jetson Orin Agx 64 GB, Jetpack: 5.1.1, Opencv + CUDA Enabled

This is my pipeline:

thetauvcsrc ! h264parse ! nvv4l2decoder ! nvvidconv ! “video/x-raw(memory:NVMM),format=RGBA” ! queue ! nvvidconv ! nvdewarper config-file=dewarp-file.txt ! m.sink_0 nvstreammux width=600 height=480 batch-size=4 num-surfaces-per-frame=4 ! nvmultistreamtiler width=1920 height=960 ! nvvidconv ! “video/x-raw(memory:NVMM),format=RGBA” ! appsink sync=0 emit-singals=true drop=true

I created a cpp file for this pipeline which is working since I confirmed it with nv3dsink instead of appsink.

Now my main aim is to get frames from nvmm memory and pass it to detectron2’s tensorrt but cv::mat uses cpu.

nvmm memory (gpu) → cv::mat (cpu) → preprocesing (cpu) → detectron2’s tensorrt (gpu)

I was wondering if there is a way I can get frames from nvmm memory and directly send it to tensorrt without using cpu. (Deepstream is perfect for this but right now, customizing deepstream for detectron2 is not something I can to do right now)

I thought of using gpumat but I am not sure it works the way I want it to.

It may be a silly question but I would appreciate any help, thanks!

3

Hi,
You may refer to the patch which uses NvBufsurface APIs + CUDA APIs to map NvBufsurface to cv::cuda::GpuMat:
How to create opencv gpumat from nvstream? - #18 by DaneLLL

1 Like
4

Hey, thanks for the reply.
I’ll check it out!
Also, I was wondering is there any sample deepstream code or something which deals with appsink?

5

Hi,
It is demonstrated in dsexample plugin. Please check the document:
Implementing a Custom GStreamer Plugin with OpenCV Integration Example — DeepStream 6.3 Release documentation

You may enable the plugin in config file and give it a try. The implementation is in get_converted_mat() in

/opt/nvidia/deepstream/deepstream-6.3/sources/gst-plugins/gst-dsexample/gstdsexample.cpp

May run the default code first and then apply the patch for a try. The way of accessing NvBufSurface is demonstrated in the plugin. It is supposed to work the same while accessing it in appsink.

1 Like
7

I verified the image by downloading it to cv::mat and seeing it for myself.
While it is RGBA, it works.

i added gst_buffer_unmap in the end which I had forgotten so it seems fine now.

Thanks!