Writing a custom Gstreamer plugin

Someone who can guide me in writing a custom GSttreamer plugin.

I want to make a custom plugin using the VPI library for dewarping.

I tried the gst-dsexample to start with, but ran into quiet a bit of problems

  • As soon as I renamed the plugin and tried to install it, it ends up in the blacklist of plugins

  • I want to use nvarguscamerasrc as the src in the stream passing NV12 data into this plugin

Any guidance and/or experience is appreciated

/Fredrik

Hi,
Leveraging dsexample is still a quick way of doing this. You can launch the piepline:

$ gst-launch-1.0 nvarguscamerasrc bufapi-version=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! mx.sink_0 nvstreammux width=1920 height=1080 batch-size=1 name=mx ! dsexample ! nvstreamdemux name=demx demx.src_0 !  nvoverlaysink

And customize dsexample to call VPI functions.

#ifdef __aarch64__
  /* To use the converted buffer in CUDA, create an EGLImage and then use
   * CUDA-EGL interop APIs */
  if (USE_EGLIMAGE) {
    if (NvBufSurfaceMapEglImage (dsexample->inter_buf, 0) !=0 ) {
      goto error;
    }

    /* dsexample->inter_buf->surfaceList[0].mappedAddr.eglImage
     * Use interop APIs cuGraphicsEGLRegisterImage and
     * cuGraphicsResourceGetMappedEglFrame to access the buffer in CUDA */

    /* Destroy the EGLImage */
    NvBufSurfaceUnMapEglImage (dsexample->inter_buf, 0);
  }
#endif

I would actually like to implement multiple plugins, so just reimplementing dsexample is a limit

1 Like

Hi,
You may add a property such as function and can set it to choose different functions:

... ! dsexample function=1 ! ...

I would really like to know how to implement specific plugins, and how to register in GStreamer.

And for my knowledge of passing data and using memory, both nvarguscamerasrc and VPI uses NVMM memory so I don’t need to convert to EGL right?

/Fredrik

Hi,
You would need to call vpiImageWrapEglImage() for hooking NvBuffer(NVMM memory) and VPIImage.

Your source is nvarguscamerasrc which is open source in r32.4.3. Maybe you can check the source code and integrate it with VPI functions. The source code is in

L4T Driver Package (BSP) Sources

Thank DaneLLL,

I will give an adaption of nvarguscamerasrc a try, feels much better to adapt closer to the source.

Just for my understanding, I thought the EGL Memory was used whenever I used the GPU. But if I use PLA features in VPI, as dewarping the memory to be used is NWM. So to wrap the image I would call
vpiImageWrapHostMem.

/Fredrik

Hi,
Did not notice there is vpiImageWrapNvBuffer(). NVMM buffer is NvBuffer so please call this function. vpiImageWrapHostMem() is to wrap CPU buffer.

Compiled and installed the libgstnvarguscamerasrc.so based on the instructions in the README.TXT fwith the source code.

Now I get tons off errors running a gstream with nvarguscamerasrc. I have made no changes to the source code.

Result when running this stream

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode

gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! “video/x-raw(memory:NVMM),width=4032,height=3040,format=NV12, framerate=$FRAMERATE/1” ! nvvidconv ! “video/x-raw(memory:NVMM),width=1000,height=754” ! nvoverlaysink overlay-w=1000 overlay-h=754

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 460: Error: Bad parameter: “ae.CameraEfficiency=0.1508”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 473: Error: Bad parameter: “ae.MeanAlg.ConvergeSpeed=0.10”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 810: Error: Bad parameter: “awb.convergeSpeed=0.3”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3100: Error: Bad parameter: “af.alg.over_scan_percent=0.15”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3101: Error: Bad parameter: “af.alg.smooth_over_scan_percent=0.035”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3107: Error: Bad parameter: “af.alg.scene_change_t_guard_factor=0.30f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3108: Error: Bad parameter: “af.alg.scene_stable_sharpness=0.15f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3109: Error: Bad parameter: “af.alg.scene_change_sharpness=0.30f”
Motion points are not strictly ascending.NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 4524: Error: 11: “tnr.v1.consistencyPoints={0.00,0.25,0.50,0.75,1.00}”
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 460: Error: Bad parameter: “ae.CameraEfficiency=0.1508”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 473: Error: Bad parameter: “ae.MeanAlg.ConvergeSpeed=0.10”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 810: Error: Bad parameter: “awb.convergeSpeed=0.3”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3100: Error: Bad parameter: “af.alg.over_scan_percent=0.15”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3101: Error: Bad parameter: “af.alg.smooth_over_scan_percent=0.035”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3107: Error: Bad parameter: “af.alg.scene_change_t_guard_factor=0.30f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3108: Error: Bad parameter: “af.alg.scene_stable_sharpness=0.15f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3109: Error: Bad parameter: “af.alg.scene_change_sharpness=0.30f”
Motion points are not strictly ascending.NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 4524: Error: 11: “tnr.v1.consistencyPoints={0.00,0.25,0.50,0.75,1.00}”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 460: Error: Bad parameter: “ae.CameraEfficiency=0.1508”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 473: Error: Bad parameter: “ae.MeanAlg.ConvergeSpeed=0.10”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 810: Error: Bad parameter: “awb.convergeSpeed=0.3”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3100: Error: Bad parameter: “af.alg.over_scan_percent=0.15”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3101: Error: Bad parameter: “af.alg.smooth_over_scan_percent=0.035”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3107: Error: Bad parameter: “af.alg.scene_change_t_guard_factor=0.30f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3108: Error: Bad parameter: “af.alg.scene_stable_sharpness=0.15f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3109: Error: Bad parameter: “af.alg.scene_change_sharpness=0.30f”
Motion points are not strictly ascending.NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 4524: Error: 11: “tnr.v1.consistencyPoints={0.00,0.25,0.50,0.75,1.00}”
GST_ARGUS: Creating output stream
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 95
Invalid default value of type ‘vec2’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 10:81:

<!-- Flow -->

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:53:

<node name=“ltmFilter” file="filters/ltm.x
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
CONSUMER: Waiting until producer is connected…
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:46:

<!-- Parameters -->
<export name="Quality" from="ltm" attribute="quality" default="3" min=

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
SCF: Error InsufficientMemory: (propagating from src/components/stages/AoltmStage.cpp, function doAsyncInitialize(), line 100)
GST_ARGUS: Available Sensor modes :
SCF: Error InsufficientMemory: AoltmStage initialization failed (in src/components/stages/AoltmStage.cpp, function doAsyncInitialize(), line 130)
GST_ARGUS: 4032 x 3040 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 22,250000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 22,250000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 4032 H = 3040
seconds to Run = 0
Frame Rate = 29,999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 115
Invalid default value of type ‘vec4’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 13:96:

<!-- Flow -->
<flow operation="execute"/>
<flow operation="detach" operand0="inputImag

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
SCF: Error InsufficientMemory: (propagating from src/components/stages/SharpenStage.cpp, function doAsyncInitialize(), line 159)
SCF: Error InsufficientMemory: SharpenStage initialization failed (in src/components/stages/SharpenStage.cpp, function doAsyncInitialize(), line 182)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 95
Invalid default value of type ‘vec2’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 10:81:

<!-- Flow -->

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:53:

<node name=“ltmFilter” file="filters/ltm42
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:49:

<!-- Parameters -->
<export name="Quality" from="ltm" attribute="quality" default="3" min=

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
SCF: Error InsufficientMemory: (propagating from src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 147)
SCF: Error InsufficientMemory: HdfxStage initialization failed (in src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 252)
CONSUMER: Producer has connected; continuing.
(Camera_ISP) Error BadParameter: Error: cct number being zero (in state_update/blocks/AP_RADTF.cpp, function NvCameraIspUpdateState_AP_RADTF_isp5(), line 107)
(Camera_ISP) Error BadParameter: (propagating from state_update/nvcamera_isp_update_state.cpp, function NvCameraIspUpdateInputStateHw(), line 100)
SCF: Error BadParameter: (propagating from src/services/autocontrol/NvCameraIspDriver.cpp, function generate(), line 1161)
SCF: Error BadParameter: (propagating from src/components/ac_stages/AcMergeStage.cpp, function doHandleRequest(), line 84)
SCF: Error BadParameter: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
SCF: Error BadParameter: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
(Camera_ISP) Error BadParameter: Error: cct number being zero (in state_update/blocks/AP_RADTF.cpp, function NvCameraIspUpdateState_AP_RADTF_isp5(), line 107)
(Camera_ISP) Error BadParameter: (propagating from state_update/nvcamera_isp_update_state.cpp, function NvCameraIspUpdateInputStateHw(), line 100)
SCF: Error BadParameter: (propagating from src/services/autocontrol/NvCameraIspDriver.cpp, function generate(), line 1161)
SCF: Error BadParameter: (propagating from src/components/ac_stages/AcMergeStage.cpp, function doHandleRequest(), line 84)
SCF: Error BadParameter: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 95
Invalid default value of type ‘vec2’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 24:81:

<!-- Flow -->
<flow operation='clone' operand0='inputImageY' operand1='outputImageY' opera

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/samples/deFog/StoreDeFog.cpp function create() line 66
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 15:57:

<node name=
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/samples/deFog/StoreDeFog.cpp function create() line 66
Out of memory
SCF: Error InsufficientMemory: (propagating from src/components/stages/DeFogStage.cpp, function doAsyncInitialize(), line 91)
SCF: Error InsufficientMemory: DeFogStage initialization failed (in src/components/stages/DeFogStage.cpp, function doAsyncInitialize(), line 151)
SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:19.347871563
Setting pipeline to PAUSED …
Setting pipeline to READY …

Hi,
Please try this patch:

Replace -largus with -largus_socketclient.

1 Like

Im a bit lost here

I assume that the ArgusLib passes a NV12 Image type in NVMM memory, right?

     imgData.type = VPI_IMAGE_TYPE_NV12;

How do I pass the image data from ArgusLib to VPI

     imgData.planes[0].data      = iFrame->getImage();

The wrapping is done by:

      CHECK_STATUS(vpiImageWrapNvBuffer(src->frameInfo->fd, 0, &vimg));

While running I get:

[ERROR] 2020-09-18 20:12:51 VPI_ERROR_INVALID_IMAGE_TYPE: Image type isn’t accepted

Hi,
The NvBuffer(s) from Argus are in block linear format and the api supports pitch linear:

Please create NvBuffer in pitch linear, call NvBufferTransform() to convert the Argus NvBuffer from blocklinear to pitchlinear, and call vpiImageWrapNvBuffer().

I still get the same error upon calling vpiImageWrapNvBuffer

  nv_mem = (GstNVArgusMemory *) mem;
  nv_mem->auxData.frame_num = consumerFrameInfo->frameNum;
  nv_mem->auxData.timestamp = consumerFrameInfo->frameTime;
  gst_mini_object_set_qdata (GST_MINI_OBJECT_CAST (buffer),
       gst_buffer_metadata_quark, &((GstNVArgusMemory *)mem)->auxData, NULL);

  retn = NvBufferTransform (consumerFrameInfo->fd, nv_mem->nvcam_buf->dmabuf_fd, &src->transform_params);


  // Dewarping
  if(src->DewarpPropSet)
  {
    // Dewarp the image

    // Wrap it into a VPIImage
    VPIImage vimg;
    {
        VPIImageData imgData;
        memset(&imgData, 0, sizeof(imgData));

      //  assert(img.type() == CV_8UC3);
        imgData.type = VPI_IMAGE_TYPE_NV12;

        // First fill VPIImageData with the, well, image data...
        imgData.numPlanes           = 1;
        imgData.planes[0].width     = src->width;
        imgData.planes[0].height    = 3040;
        imgData.planes[0].rowStride = 0; //img.step[0];
        imgData.planes[0].data      = &((GstNVArgusMemory *)mem)->auxData;

        // Now create a VPIImage that wraps it.
        CHECK_STATUS(vpiImageWrapNvBuffer(nv_mem->nvcam_buf->dmabuf_fd, 0, &vimg));
    }

The creation of the buffer is changed to pitch_linear

if (src->frameInfo->fd < 0)
{
  src->frameInfo->fd = iNativeBuffer->createNvBuffer(streamSize,
          NvBufferColorFormat_YUV420,
          NvBufferLayout_Pitch );

Hi,
Please try

  src->frameInfo->fd = iNativeBuffer->createNvBuffer(streamSize,
          NvBufferColorFormat_NV12,
          NvBufferLayout_Pitch);
  VPIImage vimg;
  vpiImageWrapNvBuffer(src->frameInfo->fd, 0, &vimg);

Thanks DaneLLL,

One step closer,

Now I get some EGL error: 12289 while running

Another question is how I feed the converted VPI image into the stream?

Third question: It seems like the vpiImageWrapNvBuffer doesn’t free up the memory

/Fredrik

Hi,
Please call vpiImageDestroy() to be paired with vpiImageWrapNvBuffer().

Suggest you implement it in consumer_thread(). The buffer sending to next element is nv_mem->nvcam_buf->dmabuf_fd. You can wrap consumerFrameInfo->fd and nv_mem->nvcam_buf->dmabuf_fd into source and destination VPI images.

Thanks, again. Really close now!

Now I get the following error

[ERROR] 2020-09-24 07:29:16 VPI_ERROR_INVALID_OPERATION: Can’t use object created in one context with device created in a different context

Coonvertion in consumer_thread()

  // Dewarping
  if(src->DewarpPropSet)
  {
    // Dewarp the image

    // Wrap the source image.
    CHECK_STATUS(vpiImageWrapNvBuffer(src->frameInfo->fd, 0, &src->m_Src_vimg));

    // Wrap the target image.
    CHECK_STATUS(vpiImageWrapNvBuffer(nv_mem->nvcam_buf->dmabuf_fd, 0, &src->m_Trgt_vimg));

    CHECK_STATUS(vpiEventRecord(src->m_evStart, src->m_stream));

    // Finaly undistorts the input image.
   CHECK_STATUS(vpiSubmitImageRemap(src->m_remap, src->m_Src_vimg, src->m_Trgt_vimg, VPI_INTERP_CATMULL_ROM, VPI_BOUNDARY_COND_ZERO));

    // Wait until conversion finishes.
   CHECK_STATUS(vpiStreamSync(src->m_stream));

    // Clean-up the wrpped images
    vpiImageDestroy( src->m_Src_vimg);
    vpiImageDestroy( src->m_Trgt_vimg);
    

  }

Initialization in gst_nv_argus_camera_src_init

/* Initialize the warping */
// Create out a vpi context to store all vpi objects we’ll create.
CHECK_STATUS(vpiContextCreate(0, &src->m_ctx));
// Activate it. From now on all created objects will be owned by it.
CHECK_STATUS(vpiContextSetCurrent(src->m_ctx));

// Create a stream for remap operation. Currently it’s implemented only on PVA.
CHECK_STATUS(vpiStreamCreate(VPI_DEVICE_TYPE_PVA, &src->m_stream));

// Now use VPI to undistort the input images:
// Temporary input and output images in NV12 format.
//CHECK_STATUS(vpiImageCreate(4032, 3040, VPI_IMAGE_TYPE_NV12, 0, &src->m_tmpIn));
//CHECK_STATUS(vpiImageCreate(4032, 3040, VPI_IMAGE_TYPE_NV12, 0, &src->m_tmpOut));

// Allocate a dense map.
src->m_map.grid.numHorizRegions = 1;
src->m_map.grid.numVertRegions = 1;
src->m_map.grid.regionWidth[0] = 4032;
src->m_map.grid.regionHeight[0] = 3040;
src->m_map.grid.horizInterval[0] = 4;
src->m_map.grid.vertInterval[0] = 4;
CHECK_STATUS(vpiWarpMapAllocData(&src->m_map));

// Initialize the fisheye lens model with the coefficients given by calibration procedure.
src->m_distModel.mapping = VPI_FISHEYE_EQUIDISTANT;
src->m_distModel.k1 = -0.126;
src->m_distModel.k2 = 0.04;
src->m_distModel.k3 = 0;
src->m_distModel.k4 = 0;

// float sensorWidth = 6.287; // IMX-477
float sensorWidth = 6.25; // IMX-477
float focalLength = 4.2;
float f = focalLength*4032/sensorWidth;
VPICameraIntrinsic K =
{
{ f, 0, 4032/2.0 },
{ 0, f, 3040/2.0 }
};
VPICameraExtrinsic X =
{
{ 1, 0, 0, 0 },
{ 0, 1, 0, 0 },
{ 0, 0, 1, 0 }
};

// Generate a warp map to undistort an image taken from fisheye lens with
// given parameters calculated above.
vpiWarpMapGenerateFromFisheyeLensDistortionModel(K, X, K, &src->m_distModel, &src->m_map);

// Create the ImageRemap payload for undistortion given the map generated above.
CHECK_STATUS(vpiCreateImageRemap(src->m_stream, &src->m_map, &src->m_remap));

// We’ll need 3 events for synchronization between the two streams above.
CHECK_STATUS(vpiEventCreate(0, &src->m_evStart));
CHECK_STATUS(vpiEventCreate(0, &src->m_ev1));
CHECK_STATUS(vpiEventCreate(0, &src->m_ev2));

I got it working by calling

   // Activate it. From now on all created objects will be owned by it.
    CHECK_STATUS(vpiContextSetCurrent(src->m_ctx));

In the consumer_thread

Now I found some performance issues,

The wrapping of the two images take the same time as the dewarping itself

Aprox: 7ms per wrapping x 2 + aprox 16ms dewarping → aprox 30ms convertion @4032x3040

Aprox: 6ms per wrapping x 2 + aprox 6 ms dewarping → aprox 18ms convertion @1920x1080

/Fredrik

Hi,
PVA is in 400MHz in default mode. Please try other modes in the table.

My Xavier NX is running in Mode2 during these tests