Nvarguscamerasrc error when using nvdewarper plugin

Please provide complete information as applicable to your setup.

• Hardware Platform Jetson Xavier NX
• DeepStream Version 5.1
• JetPack Version 4.5.1
• TensorRT Version N/A
• NVIDIA GPU Driver Version N/A
• Issue Type bug?
• How to reproduce the issue ? See Below

We are trying to run a live camera stream through the nvdewarper plugin using the perspective projection method, but we cannot get past a particular SIGSEV that seems to be related to errors with the nvargus-daemon service. We are working on a Jetson Xavier NX running jetpack 4.5.1, deepstream 5.1, and an FPDLINK III camerahan IMX335 sensor.

The Pipeline:
gst-launch-1.0 -v nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=(int)2616, height=(int)1946, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)2616, height=(int)1984, format=(string)RGBA, framerate=(fraction)30/1' ! nvdewarper config-file=[FILEPATH] source-id=0 ! m.sink_0 nvstreammux name=m width=2616 height=1984 batch-size=1 num-surfaces-per-frame=1 ! nvegltransform ! nveglglessink -e

The config file:

[property]
#dewarp-dump-frames=10
output-width=2616
output-height=1984
num-batch-buffers=1

[surface0]
# 1=PushBroom, 2=VertRadCyl, 3=PerspectivePerspective
projection-type=3
width=2616
height=1984
focal-length=1721.592
#src-fov=180
#top-angle=30
#bottom-angle=-30
distortion=-0.09787;0.00574;-0.00013;0;0
# 0=cudaAddressModeClamp, 1=cudaAddressModeBorder
#cuda-address-mode=0

The Gstreamer output:

Setting pipeline to PAUSED …

Using winsys: x11 
Pipeline is live and does not need PREROLL ...
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1946, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1946, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1984, framerate=(fraction)30/1, format=(string)RGBA, pixel-aspect-ratio=(fraction)992/973
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1984, framerate=(fraction)30/1, format=(string)RGBA, pixel-aspect-ratio=(fraction)992/973
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)RGBA, framerate=(fraction)30/1, width=(int)2616, height=(int)1984
/GstPipeline:pipeline0/GstNvStreamMux:m.GstPad:src: caps = video/x-raw(memory:NVMM), format=(string)RGBA, framerate=(fraction)30/1, width=(int)2616, height=(int)1984, batch-size=(int)1, num-surfaces-per-frame=(int)1
/GstPipeline:pipeline0/GstNvEglTransform:nvegltransform0.GstPad:src: caps = video/x-raw(memory:EGLImage), format=(string)RGBA, framerate=(fraction)30/1, width=(int)2616, height=(int)1984, batch-size=(int)1, num-surfaces-per-frame=(int)1
/GstPipeline:pipeline0/GstEglGlesSink:eglglessink0.GstPad:sink: caps = video/x-raw(memory:EGLImage), format=(string)RGBA, framerate=(fraction)30/1, width=(int)2616, height=(int)1984, batch-size=(int)1, num-surfaces-per-frame=(int)1
/GstPipeline:pipeline0/GstNvEglTransform:nvegltransform0.GstPad:sink: caps = video/x-raw(memory:NVMM), format=(string)RGBA, framerate=(fraction)30/1, width=(int)2616, height=(int)1984, batch-size=(int)1, num-surfaces-per-frame=(int)1
/GstPipeline:pipeline0/GstNvStreamMux:m.GstNvStreamPad:sink_0: caps = video/x-raw(memory:NVMM), format=(string)RGBA, framerate=(fraction)30/1, width=(int)2616, height=(int)1984
/GstPipeline:pipeline0/Gstnvdewarper:nvdewarper0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1984, framerate=(fraction)30/1, format=(string)RGBA, pixel-aspect-ratio=(fraction)992/973
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1984, framerate=(fraction)30/1, format=(string)RGBA, pixel-aspect-ratio=(fraction)992/973
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1946, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)2616, height=(int)1946, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 2616 x 1946 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 15000, max 16650000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 2616 H = 1946 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Caught SIGSEGV
#0  0x0000007f7ccd0e28 in __GI___poll (fds=0x559f346710, nfds=547555763080, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000007f7cdddf58 in  () at /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#2  0x000000559e89fba0 in  ()
Spinning.  Please run 'gdb gst-launch-1.0 9245' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

The output of sudo enableCamInfiniteTimeout=1 nvargus-daemon throws the following errors:

=== NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== gst-launch-1.0[9245]: Connection established (7FA08271D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 390: Error: Invalid isp config attribute element: "mwbCCT.sunlight=6100"
NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 393: Error: Invalid isp config attribute element: "mwbCCT.tungsten=2950"
NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 396: Error: Invalid isp config attribute element: "mwbCCT.flash=7300"
NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 397: Error: Invalid isp config attribute element: "mwbCCT.horizon=3000"
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[9245]: CameraProvider initialized (0x7f9c826640)NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 390: Error: Invalid isp config attribute element: "mwbCCT.sunlight=6100"
NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 393: Error: Invalid isp config attribute element: "mwbCCT.tungsten=2950"
NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 396: Error: Invalid isp config attribute element: "mwbCCT.flash=7300"
NvCameraIspConfigFileLoad: type = 0x75 Config file "camera_overrides.isp" Line 397: Error: Invalid isp config attribute element: "mwbCCT.horizon=3000"
CAM: serial no file already exists, skips storing againNvIspAfConfigParamsSanityCheck: Error: positionWorkingHigh is not larger than positionWorkingLow positionWorkingHigh = 0, positionWorkingLow = 0

NvIspAfConfigParamsSanityCheck: Error: positionWorkingHigh is not larger than positionWorkingLow positionWorkingHigh = 0, positionWorkingLow = 0

NvIspAfConfigParamsSanityCheck: Error: positionWorkingHigh is not larger than positionWorkingLow positionWorkingHigh = 0, positionWorkingLow = 0

SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function returnFrame(), line 372)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function getBuffer(), line 434)
SCF: Error InvalidState:  (propagating from src/components/CaptureContainerImpl.cpp, function assignAllBuffersFromStream(), line 230)
SCF: Error InvalidState:  (propagating from src/components/stages/CCDataSetupStage.cpp, function doHandleRequest(), line 68)
SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 992)
=== gst-launch-1.0[9245]: Connection closed (7FA08271D0)=== gst-launch-1.0[9245]: WARNING: CameraProvider was not destroyed before client connection terminated.=== gst-launch-1.0[9245]:          The client may have abnormally terminated. Destroying CameraProvider...=== gst-launch-1.0[9245]: CameraProvider destroyed (0x7f9c826640)=== gst-launch-1.0[9245]: WARNING: Cleaning up 1 outstanding requests...=== gst-launch-1.0[9245]: WARNING: Cleaning up 1 outstanding stream settings...=== gst-launch-1.0[9245]: WARNING: Cleaning up 1 outstanding queues...=== gst-launch-1.0[9245]: WARNING: Cleaning up 1 outstanding sessions...

We would appreciate any help that can be given here.

Thanks!

Hi,
For using nvarguscamerasrc in DeepStream SDK, please set the property:

  bufapi-version      : set to use new Buffer API
                        flags: readable, writable
                        Boolean. Default: false

And use nvvideoocnvert plugin instead of nvvidconv.

We really appreciate your feedback DaneLLL - this change does produce a working pipeline, but the output framerate is very poor (<1 FPS). Are there significant optimizations we are missing here? Or does it make more sense to leverage OpenCV in gst-dsexample to get a reasonable livestreamed framerate rather than spending time working with this plugin?

The above changes result in the following warning messages from gstreamer:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:eglglessink0:
There may be a timestamping problem, or this computer is too slow.

EDIT: Nevermind, we removed the nvstreammux from the pipeline and are seeing excellent performance now. I will leave this up in case someone else runs into the same issue. Thanks for your help Dane!

Hi,
If you don’t have audio source, we would suggest set sync=0 to nveglglessink plugin. Since the camera source is a live source in steady 30fps, it is fine to disable synchronization mechanism in gstreamer.

Thanks Dane, that did improve the pipeline latency by about 20%. We are still above 100 ms and are looking into any available optimizations.