Writing a custom Gstreamer plugin

Someone who can guide me in writing a custom GSttreamer plugin.

I want to make a custom plugin using the VPI library for dewarping.

I tried the gst-dsexample to start with, but ran into quiet a bit of problems

  • As soon as I renamed the plugin and tried to install it, it ends up in the blacklist of plugins

  • I want to use nvarguscamerasrc as the src in the stream passing NV12 data into this plugin

Any guidance and/or experience is appreciated

/Fredrik

Hi,
Leveraging dsexample is still a quick way of doing this. You can launch the piepline:

$ gst-launch-1.0 nvarguscamerasrc bufapi-version=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=30/1' ! mx.sink_0 nvstreammux width=1920 height=1080 batch-size=1 name=mx ! dsexample ! nvstreamdemux name=demx demx.src_0 !  nvoverlaysink

And customize dsexample to call VPI functions.

#ifdef __aarch64__
  /* To use the converted buffer in CUDA, create an EGLImage and then use
   * CUDA-EGL interop APIs */
  if (USE_EGLIMAGE) {
    if (NvBufSurfaceMapEglImage (dsexample->inter_buf, 0) !=0 ) {
      goto error;
    }

    /* dsexample->inter_buf->surfaceList[0].mappedAddr.eglImage
     * Use interop APIs cuGraphicsEGLRegisterImage and
     * cuGraphicsResourceGetMappedEglFrame to access the buffer in CUDA */

    /* Destroy the EGLImage */
    NvBufSurfaceUnMapEglImage (dsexample->inter_buf, 0);
  }
#endif

I would actually like to implement multiple plugins, so just reimplementing dsexample is a limit

Hi,
You may add a property such as function and can set it to choose different functions:

... ! dsexample function=1 ! ...

I would really like to know how to implement specific plugins, and how to register in GStreamer.

And for my knowledge of passing data and using memory, both nvarguscamerasrc and VPI uses NVMM memory so I don’t need to convert to EGL right?

/Fredrik

Hi,
You would need to call vpiImageWrapEglImage() for hooking NvBuffer(NVMM memory) and VPIImage.

Your source is nvarguscamerasrc which is open source in r32.4.3. Maybe you can check the source code and integrate it with VPI functions. The source code is in


L4T Driver Package (BSP) Sources

Thank DaneLLL,

I will give an adaption of nvarguscamerasrc a try, feels much better to adapt closer to the source.

Just for my understanding, I thought the EGL Memory was used whenever I used the GPU. But if I use PLA features in VPI, as dewarping the memory to be used is NWM. So to wrap the image I would call
vpiImageWrapHostMem.

/Fredrik

Hi,
Did not notice there is vpiImageWrapNvBuffer(). NVMM buffer is NvBuffer so please call this function. vpiImageWrapHostMem() is to wrap CPU buffer.

Compiled and installed the libgstnvarguscamerasrc.so based on the instructions in the README.TXT fwith the source code.

Now I get tons off errors running a gstream with nvarguscamerasrc. I have made no changes to the source code.

Result when running this stream

SENSOR_ID=0 # 0 for CAM0 and 1 for CAM1 ports
FRAMERATE=30 # Framerate can go from 2 to 30 for 4032x3040 mode

gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! “video/x-raw(memory:NVMM),width=4032,height=3040,format=NV12, framerate=$FRAMERATE/1” ! nvvidconv ! “video/x-raw(memory:NVMM),width=1000,height=754” ! nvoverlaysink overlay-w=1000 overlay-h=754

Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 460: Error: Bad parameter: “ae.CameraEfficiency=0.1508”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 473: Error: Bad parameter: “ae.MeanAlg.ConvergeSpeed=0.10”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 810: Error: Bad parameter: “awb.convergeSpeed=0.3”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3100: Error: Bad parameter: “af.alg.over_scan_percent=0.15”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3101: Error: Bad parameter: “af.alg.smooth_over_scan_percent=0.035”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3107: Error: Bad parameter: “af.alg.scene_change_t_guard_factor=0.30f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3108: Error: Bad parameter: “af.alg.scene_stable_sharpness=0.15f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3109: Error: Bad parameter: “af.alg.scene_change_sharpness=0.30f”
Motion points are not strictly ascending.NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 4524: Error: 11: “tnr.v1.consistencyPoints={0.00,0.25,0.50,0.75,1.00}”
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 460: Error: Bad parameter: “ae.CameraEfficiency=0.1508”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 473: Error: Bad parameter: “ae.MeanAlg.ConvergeSpeed=0.10”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 810: Error: Bad parameter: “awb.convergeSpeed=0.3”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3100: Error: Bad parameter: “af.alg.over_scan_percent=0.15”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3101: Error: Bad parameter: “af.alg.smooth_over_scan_percent=0.035”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3107: Error: Bad parameter: “af.alg.scene_change_t_guard_factor=0.30f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3108: Error: Bad parameter: “af.alg.scene_stable_sharpness=0.15f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3109: Error: Bad parameter: “af.alg.scene_change_sharpness=0.30f”
Motion points are not strictly ascending.NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 4524: Error: 11: “tnr.v1.consistencyPoints={0.00,0.25,0.50,0.75,1.00}”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 460: Error: Bad parameter: “ae.CameraEfficiency=0.1508”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 473: Error: Bad parameter: “ae.MeanAlg.ConvergeSpeed=0.10”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 810: Error: Bad parameter: “awb.convergeSpeed=0.3”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3100: Error: Bad parameter: “af.alg.over_scan_percent=0.15”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3101: Error: Bad parameter: “af.alg.smooth_over_scan_percent=0.035”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3107: Error: Bad parameter: “af.alg.scene_change_t_guard_factor=0.30f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3108: Error: Bad parameter: “af.alg.scene_stable_sharpness=0.15f”
NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 3109: Error: Bad parameter: “af.alg.scene_change_sharpness=0.30f”
Motion points are not strictly ascending.NvCameraIspConfigFileLoad: type = 0x5 Config file “common.cfg” Line 4524: Error: 11: “tnr.v1.consistencyPoints={0.00,0.25,0.50,0.75,1.00}”
GST_ARGUS: Creating output stream
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 95
Invalid default value of type ‘vec2’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 10:81:

<!-- Flow -->

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:53:

<node name=“ltmFilter” file="filters/ltm.x
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
CONSUMER: Waiting until producer is connected…
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:46:

<!-- Parameters -->
<export name="Quality" from="ltm" attribute="quality" default="3" min=

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
SCF: Error InsufficientMemory: (propagating from src/components/stages/AoltmStage.cpp, function doAsyncInitialize(), line 100)
GST_ARGUS: Available Sensor modes :
SCF: Error InsufficientMemory: AoltmStage initialization failed (in src/components/stages/AoltmStage.cpp, function doAsyncInitialize(), line 130)
GST_ARGUS: 4032 x 3040 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 22,250000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 22,250000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 4032 H = 3040
seconds to Run = 0
Frame Rate = 29,999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 115
Invalid default value of type ‘vec4’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 13:96:

<!-- Flow -->
<flow operation="execute"/>
<flow operation="detach" operand0="inputImag

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
SCF: Error InsufficientMemory: (propagating from src/components/stages/SharpenStage.cpp, function doAsyncInitialize(), line 159)
SCF: Error InsufficientMemory: SharpenStage initialization failed (in src/components/stages/SharpenStage.cpp, function doAsyncInitialize(), line 182)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 95
Invalid default value of type ‘vec2’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 10:81:

<!-- Flow -->

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:53:

<node name=“ltmFilter” file="filters/ltm42
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 6:49:

<!-- Parameters -->
<export name="Quality" from="ltm" attribute="quality" default="3" min=

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/storeHDFX/StoreHDFX.cpp function create() line 92
Out of memory
SCF: Error InsufficientMemory: (propagating from src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 147)
SCF: Error InsufficientMemory: HdfxStage initialization failed (in src/components/stages/HdfxStage.cpp, function doAsyncInitialize(), line 252)
CONSUMER: Producer has connected; continuing.
(Camera_ISP) Error BadParameter: Error: cct number being zero (in state_update/blocks/AP_RADTF.cpp, function NvCameraIspUpdateState_AP_RADTF_isp5(), line 107)
(Camera_ISP) Error BadParameter: (propagating from state_update/nvcamera_isp_update_state.cpp, function NvCameraIspUpdateInputStateHw(), line 100)
SCF: Error BadParameter: (propagating from src/services/autocontrol/NvCameraIspDriver.cpp, function generate(), line 1161)
SCF: Error BadParameter: (propagating from src/components/ac_stages/AcMergeStage.cpp, function doHandleRequest(), line 84)
SCF: Error BadParameter: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
SCF: Error BadParameter: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)
(Camera_ISP) Error BadParameter: Error: cct number being zero (in state_update/blocks/AP_RADTF.cpp, function NvCameraIspUpdateState_AP_RADTF_isp5(), line 107)
(Camera_ISP) Error BadParameter: (propagating from state_update/nvcamera_isp_update_state.cpp, function NvCameraIspUpdateInputStateHw(), line 100)
SCF: Error BadParameter: (propagating from src/services/autocontrol/NvCameraIspDriver.cpp, function generate(), line 1161)
SCF: Error BadParameter: (propagating from src/components/ac_stages/AcMergeStage.cpp, function doHandleRequest(), line 84)
SCF: Error BadParameter: (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/value.cpp function set() line 95
Invalid default value of type ‘vec2’
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/attribute.h function setDefault() line 452
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function handleElementAttribute() line 666
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function handleElementFilter() line 73
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/filters/filterUser.cpp function xmlHandleStartElementFilter() line 49
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 24:81:

<!-- Flow -->
<flow operation='clone' operand0='inputImageY' operand1='outputImageY' opera

FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/samples/deFog/StoreDeFog.cpp function create() line 66
Out of memory
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function load() line 317
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function handleElementNetwork() line 103
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/networks/networkUser.cpp function xmlHandleStartElementNetwork() line 52
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/xmlParse.cpp function parse() line 146
parsing aborted at line 15:57:

<node name=
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 339
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/src/common/node.cpp function create() line 41
(propagating)
FNET: Error in /dvs/git/dirty/git-master_linux/camera/utils/nvfnet/samples/deFog/StoreDeFog.cpp function create() line 66
Out of memory
SCF: Error InsufficientMemory: (propagating from src/components/stages/DeFogStage.cpp, function doAsyncInitialize(), line 91)
SCF: Error InsufficientMemory: DeFogStage initialization failed (in src/components/stages/DeFogStage.cpp, function doAsyncInitialize(), line 151)
SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:19.347871563
Setting pipeline to PAUSED …
Setting pipeline to READY …

Hi,
Please try this patch:


Replace -largus with -largus_socketclient.