[JP4.4][BUG]: nvargus-daemon SIGSEGV after exit is repeatable

Problem:
After I stop a Gstreamer python app with CTRL-C from userspace using nvarguscamerasrc on sendor-id=0 I got always a segmentation fault.

OS:

R32 (release), REVISION: 4.2, GCID: 20074772, BOARD: t210ref, EABI: aarch64, DATE: Thu Apr 9 01:22:12 UTC 2020

Software:

openCV 4.3.0+Cuda 10.2 + cuDNN 7.6.3

Gstreamer setup:

"nvarguscamerasrc num-buffers=1024 sensor-id=%d ! video/x-raw(memory:NVMM), width=(int)%d, height=(int)%d, format=(string)NV12, framerate=(fraction)%d/1 ! nvvidconv flip-method=%d ! video/x-raw, display_width=(int)%d, display_height=(int)%d, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink "

Installed packages:

$ dpkg -l | grep -E “nvidia|cuda-nv|libnv” | awk ‘{print $1,$2,$3}’
ii cuda-nvcc-10-0 10.0.326-1
ii cuda-nvcc-10-2 10.2.89-1
ii cuda-nvdisasm-10-0 10.0.326-1
ii cuda-nvdisasm-10-2 10.2.89-1
ii cuda-nvgraph-10-0 10.0.326-1
ii cuda-nvgraph-10-2 10.2.89-1
ii cuda-nvgraph-dev-10-0 10.0.326-1
ii cuda-nvgraph-dev-10-2 10.2.89-1
ii cuda-nvml-dev-10-0 10.0.326-1
ii cuda-nvml-dev-10-2 10.2.89-1
ii cuda-nvprof-10-0 10.0.326-1
ii cuda-nvprof-10-2 10.2.89-1
ii cuda-nvprune-10-0 10.0.326-1
ii cuda-nvprune-10-2 10.2.89-1
ii cuda-nvrtc-10-0 10.0.326-1
ii cuda-nvrtc-10-2 10.2.89-1
ii cuda-nvrtc-dev-10-0 10.0.326-1
ii cuda-nvrtc-dev-10-2 10.2.89-1
ii cuda-nvtx-10-0 10.0.326-1
ii cuda-nvtx-10-2 10.2.89-1
ii libnvidia-container-tools 0.9.0~beta.1
ii libnvidia-container0:arm64 0.9.0~beta.1
ii libnvinfer-bin 7.1.0-1+cuda10.2
ii libnvinfer-dev 7.1.0-1+cuda10.2
ii libnvinfer-doc 7.1.0-1+cuda10.2
ii libnvinfer-plugin-dev 7.1.0-1+cuda10.2
ii libnvinfer-plugin6 6.0.1-1+cuda10.0
ii libnvinfer-plugin7 7.1.0-1+cuda10.2
ii libnvinfer-samples 7.1.0-1+cuda10.2
ii libnvinfer6 6.0.1-1+cuda10.0
ii libnvinfer7 7.1.0-1+cuda10.2
ii libnvonnxparsers-dev 7.1.0-1+cuda10.2
ii libnvonnxparsers6 6.0.1-1+cuda10.0
ii libnvonnxparsers7 7.1.0-1+cuda10.2
ii libnvparsers-dev 7.1.0-1+cuda10.2
ii libnvparsers6 6.0.1-1+cuda10.0
ii libnvparsers7 7.1.0-1+cuda10.2
ii nvidia-container-csv-cuda 10.2.89-1
ii nvidia-container-csv-cudnn 8.0.0.145-1+cuda10.2
ii nvidia-container-csv-tensorrt 7.1.0.16-1+cuda10.2
ii nvidia-container-csv-visionworks 1.6.0.501
ii nvidia-container-runtime 3.1.0-1
ii nvidia-container-toolkit 1.0.1-1
ii nvidia-docker2 2.2.0-1
ii nvidia-l4t-3d-core 32.4.2-20200408182156
ii nvidia-l4t-apt-source 32.4.2-20200408182156
ii nvidia-l4t-bootloader 32.4.2-20200408182156
ii nvidia-l4t-camera 32.4.2-20200408182156
rc nvidia-l4t-ccp-t210ref 32.3.1-20191209225816
ii nvidia-l4t-configs 32.4.2-20200408182156
ii nvidia-l4t-core 32.4.2-20200408182156
ii nvidia-l4t-cuda 32.4.2-20200408182156
ii nvidia-l4t-firmware 32.4.2-20200408182156
ii nvidia-l4t-graphics-demos 32.4.2-20200408182156
ii nvidia-l4t-gstreamer 32.4.2-20200423124921
ii nvidia-l4t-init 32.4.2-20200408182156
ii nvidia-l4t-initrd 32.4.2-20200408182156
ii nvidia-l4t-jetson-io 32.4.2-20200408182156
ii nvidia-l4t-jetson-multimedia-api 32.4.2-20200408182156
ii nvidia-l4t-kernel 4.9.140-tegra-32.4.2-20200408182156
ii nvidia-l4t-kernel-dtbs 4.9.140-tegra-32.4.2-20200408182156
ii nvidia-l4t-kernel-headers 4.9.140-tegra-32.4.2-20200408182156
ii nvidia-l4t-multimedia 32.4.2-20200408182156
ii nvidia-l4t-multimedia-utils 32.4.2-20200408182156
ii nvidia-l4t-oem-config 32.4.2-20200408182156
ii nvidia-l4t-tools 32.4.2-20200408182156
ii nvidia-l4t-wayland 32.4.2-20200408182156
ii nvidia-l4t-weston 32.4.2-20200408182156
ii nvidia-l4t-x11 32.4.2-20200408182156
ii nvidia-l4t-xusb-firmware 32.4.2-20200408182156
ii python-libnvinfer 7.1.0-1+cuda10.2
ii python-libnvinfer-dev 7.1.0-1+cuda10.2
ii python3-libnvinfer 7.1.0-1+cuda10.2
ii python3-libnvinfer-dev 7.1.0-1+cuda10.2

Termination error.

^C terminate called without an active exception
Aborted (core dumped)

Start error:

Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:568 Failed to create CaptureSession

Question:

  1. Is there a fix to start the app without restarting nvargus-daemon? Keep in mind the python app runs as a docker container?
  2. Is there a way to talk and sniff the argus_socket (just for testing if it is running and issue a restart from the docker container itself)?
  3. How can I restart the daemon from the python app itself?

Hi,
We don’t observe the issue on Jetson Nano + Raspberry Pi camera V2. Probably it is specific to your camera source. For information, please share which camera board you use.

My camera is a raspberry PI Camera Module V2

$ gst-inspect-1.0 nvarguscamerasrc
Factory Details:
  Rank                     primary (256)
  Long-name                NvArgusCameraSrc
  Klass                    Video/Capture
  Description              nVidia ARGUS Camera Source
  Author                   Viranjan Pagar <vpagar@nvidia.com>, Amit Pandya <apandya@nvidia.com>

Plugin Details:
  Name                     nvarguscamerasrc
  Description              nVidia ARGUS Source Component
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so
  Version                  1.0.0
  License                  Proprietary
  Source module            nvarguscamerasrc
  Binary package           NvARGUSCameraSrc
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstNvArgusCameraSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 format: { (string)NV12 }
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvarguscamerasrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  typefind            : Run typefind before negotiating (deprecated, non-functional)
                        flags: readable, writable, deprecated
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: true
  silent              : Produce verbose output ?
                        flags: readable, writable
                        Boolean. Default: true
  timeout             : timeout to capture in seconds (Either specify timeout or num-buffers, not both)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0 
  wbmode              : White balance affects the color temperature of the photo
                        flags: readable, writable
                        Enum "GstNvArgusCamWBMode" Default: 1, "auto"
                           (0): off              - GST_NVCAM_WB_MODE_OFF
                           (1): auto             - GST_NVCAM_WB_MODE_AUTO
                           (2): incandescent     - GST_NVCAM_WB_MODE_INCANDESCENT
                           (3): fluorescent      - GST_NVCAM_WB_MODE_FLUORESCENT
                           (4): warm-fluorescent - GST_NVCAM_WB_MODE_WARM_FLUORESCENT
                           (5): daylight         - GST_NVCAM_WB_MODE_DAYLIGHT
                           (6): cloudy-daylight  - GST_NVCAM_WB_MODE_CLOUDY_DAYLIGHT
                           (7): twilight         - GST_NVCAM_WB_MODE_TWILIGHT
                           (8): shade            - GST_NVCAM_WB_MODE_SHADE
                           (9): manual           - GST_NVCAM_WB_MODE_MANUAL
  saturation          : Property to adjust saturation value
                        flags: readable, writable
                        Float. Range:               0 -               2 Default:               1 
  sensor-id           : Set the id of camera sensor to use. Default 0.
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 0 
  sensor-mode         : Set the camera sensor mode to use. Default -1 (Select the best match)
                        flags: readable, writable
                        Integer. Range: -1 - 255 Default: -1 
  total-sensor-modes  : Query the number of sensor modes available. Default 0
                        flags: readable
                        Integer. Range: 0 - 255 Default: 0 
  exposuretimerange   : Property to adjust exposure time range in nanoseconds
                        Use string with values of Exposure Time Range (low, high)
                        in that order, to set the property.
                        eg: exposuretimerange="34000 358733000"
                        flags: readable, writable
                        String. Default: null
  gainrange           : Property to adjust gain range
                        Use string with values of Gain Time Range (low, high)
                        in that order, to set the property.
                        eg: gainrange="1 16"
                        flags: readable, writable
                        String. Default: null
  ispdigitalgainrange : Property to adjust digital gain range
                        Use string with values of ISP Digital Gain Range (low, high)
                        in that order, to set the property.
                        eg: ispdigitalgainrange="1 8"
                        flags: readable, writable
                        String. Default: null
  tnr-strength        : property to adjust temporal noise reduction strength
                        flags: readable, writable
                        Float. Range:              -1 -               1 Default:              -1 
  tnr-mode            : property to select temporal noise reduction mode
                        flags: readable, writable
                        Enum "GstNvArgusCamTNRMode" Default: 1, "NoiseReduction_Fast"
                           (0): NoiseReduction_Off - GST_NVCAM_NR_OFF
                           (1): NoiseReduction_Fast - GST_NVCAM_NR_FAST
                           (2): NoiseReduction_HighQuality - GST_NVCAM_NR_HIGHQUALITY
  ee-mode             : property to select edge enhnacement mode
                        flags: readable, writable
                        Enum "GstNvArgusCamEEMode" Default: 1, "EdgeEnhancement_Fast"
                           (0): EdgeEnhancement_Off - GST_NVCAM_EE_OFF
                           (1): EdgeEnhancement_Fast - GST_NVCAM_EE_FAST
                           (2): EdgeEnhancement_HighQuality - GST_NVCAM_EE_HIGHQUALITY
  ee-strength         : property to adjust edge enhancement strength
                        flags: readable, writable
                        Float. Range:              -1 -               1 Default:              -1 
  aeantibanding       : property to set the auto exposure antibanding mode
                        flags: readable, writable
                        Enum "GstNvArgusCamAeAntiBandingMode" Default: 1, "AeAntibandingMode_Auto"
                           (0): AeAntibandingMode_Off - GST_NVCAM_AEANTIBANDING_OFF
                           (1): AeAntibandingMode_Auto - GST_NVCAM_AEANTIBANDING_AUTO
                           (2): AeAntibandingMode_50HZ - GST_NVCAM_AEANTIBANDING_50HZ
                           (3): AeAntibandingMode_60HZ - GST_NVCAM_AEANTIBANDING_60HZ
  exposurecompensation: property to adjust exposure compensation
                        flags: readable, writable
                        Float. Range:              -2 -               2 Default:               0 
  aelock              : set or unset the auto exposure lock
                        flags: readable, writable
                        Boolean. Default: false
  awblock             : set or unset the auto white balance lock
                        flags: readable, writable
                        Boolean. Default: false
  bufapi-version      : set to use new Buffer API
                        flags: readable, writable
                        Boolean. Default: false

Do you have the same camera and tested with r32-4.2 ?

Keep in mind I have the first board release:

JETSON_BOARD='P3448-0000 P3449-0000'
JETSON_BOARDIDS=3448
JETSON_CHIP_ID=33

Seems you have cudnn 8.0 while you mentionned 7.6.3. Be sure there is no wrong setting with this.

You may also try to use EOS (-e with gst-launch) and monitor with:

tail -f /var/log/syslog

Right I double checked everything and I am shure to have 7.6.3

$ ls -las /etc/alternatives/cudnn| awk ‘{print $10,$11,$12}’ | grep 7
/usr/include/aarch64-linux-gnu/cudnn.h → /etc/alternatives/cudnn_h
/etc/alternatives/cudnn_h → /usr/include/aarch64-linux-gnu/cudnn_v7.h
/etc/alternatives/libcudnn_so → /usr/lib/aarch64-linux-gnu/libcudnn.so.7

$ ls -las /usr/lib/aarch64-linux-gnu/*cudnn* | awk '{print $10,$11,$12}' |grep 7
/usr/lib/aarch64-linux-gnu/libcudnn.so -> /etc/alternatives/libcudnn_so
/usr/lib/aarch64-linux-gnu/libcudnn.so.7 -> libcudnn.so.7.6.3
/usr/lib/aarch64-linux-gnu/libcudnn.so.7.6.3  
/usr/lib/aarch64-linux-gnu/libcudnn_static_v7.a

I also noticed similar problem when closing opencv application with SIG_TERM, but it was only happening after some problematic OTA upgrade from L4T R32.3.1 to R32.4.2…took some time to auto clean up but seems fixed now.
I used a solution that @Andrey1984 shared here for this case in C++… If it persists, you may try to translate and test with python.

Hi,

We have verified the python script:

import sys
import cv2

def read_cam():
    cap = cv2.VideoCapture("nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080,format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert !  appsink")
    if cap.isOpened():
        cv2.namedWindow("demo", cv2.WINDOW_AUTOSIZE)
        while True:
            ret_val, img = cap.read();
            cv2.imshow('demo',img)
            cv2.waitKey(10)
    else:
     print "camera open failed"

    cv2.destroyAllWindows()


if __name__ == '__main__':
    read_cam()

No issue is seen. Please give it a try.

Hi DaneLLL thx for your help.

Thx Honey_Patouceul, I could fix some of my problems so far.

My code has one thread for a flask server AND another for the videostream. I had to monitor the main thread termination in the videostream thread break the loop and call
vc.release()
cv2.destroyAllWindows()

This is what I get after successful termination :)

[INFO] Inspector thread will exit...
[INFO] cleaning up...
GST_ARGUS: Cleaning up

(python3:21947): GStreamer-CRITICAL **: 23:38:25.825: gst_mini_object_set_qdata: assertion 'object != NULL' failed

(python3:21947): GStreamer-CRITICAL **: 23:38:25.825: gst_mini_object_set_qdata: assertion 'object != NULL' failed
CONSUMER: Done Success
GST_ARGUS: Done Success

The last two lines shows correct thread termination with KeyboardInterrupt or SystemExit