R32.3.1 tx2-4g gst_element_change_state hangs

Hi,
Please send EoS before switching to NULL state, and unref, re-initialize the pipeline. The sample code of termination:

    // Send EoS(End of Stream)
    gst_element_send_event ((GstElement*)gst_pipeline, gst_event_new_eos ());
    // Wait for EOS message
    bus = gst_pipeline_get_bus(GST_PIPELINE(gst_pipeline));
    gst_bus_poll(bus, GST_MESSAGE_EOS, GST_CLOCK_TIME_NONE);
    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);

    gst_object_unref(GST_OBJECT(gst_pipeline));

It helps that you provide the pipeline.
I’m unable to fully try because there are custom plugins such as stalkertakepicture that I don’t have, but I see at least 2 problems with it:

  1. The crop parameters and output are not correct (reread this). So change the crop parameters so that
    [EDIT: needs to be corrected by -1 wrt to output resolution
    right = left + 480 -1 and bottom = top + 380 -1]:
nvvidconv name=stalkerLoc flip-method=2 top=520 bottom899 left=1212 right=1691 \
     ! 'video/x-raw(memory:NVMM),width=480,height=380'

or, if your ROI is 1440x1140 and just want to rescale for display:

nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1212 right=2651 \
     ! 'video/x-raw(memory:NVMM),width=1440,height=1140' \
     ! nvvidconv \
     ! 'video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1'
  1. For using nvjpegenc, you need to have I420 format in NVMM memory:
nvvidconv \
      ! 'video/x-raw(memory:NVMM),format=I420' \
      ! nvjpegenc 

make stalkerTakePicture and stalkerCreateVideo valve, instead

Also the display is working and the jpg is being created correctly.

Will look at this after I get the app to close down correctly and restart correctly.

Thanks,
Terry

@DaneLLL
Thanks for the code snippet. I put it in and my app hangs on the gst_bus_poll, how do I debug that please

remember the pipeline is in the PAUSED state, and the pending is PLAYING.

I have put all element in the NULL state, but it did not change the pipeline state.

So my pipeline is not behaving correctly, so what do I need to look at next. Because the set_state to NULL hangs, I commented it out. looks like gst_bus_poll is hanging also,

Terry

Ok, I’ll try that, but I’ll be away from any Jetson till next week.

Maybe it can work…however for a solution that could work on various recent L4T releases, I’d suggest to be sure.

Be also aware that stopping a wrong pipeline may also trigger issues. Maybe it would be easier to stop a correctly working pipeline…some issues may only happen at close time.

I also notice from your logs that one plugin, stalkercreatevideo, returns different code and status when you’re trying to stop:

gstinterface.cpp [1081] Stop() :” stalkerVideoSink 0x55a0808030 sret 2 state 2 Pending 4

Sorry I don’t know the code and enums for your logs, but this discrepancy would make wonder if it fails without this plugin. Does using valve instead show similar error ?

I’ve further looked into your issue.

Further than the above advice:

Not sure about the odds that a fix for R32.3.1 can be soon provided, but this will hopefully help next releases to be fixed.

@DaneLLL Any feed back about my sending EOS hanging waiting for it to return?

EOS does not terminate my stream, and setting it to NULL does not terminate the stream.

So I kill the app and get a bunch of these messages.

(VideoSystem:10224): GStreamer-CRITICAL **: 15:49:55.237:
Trying to dispose element capsfilter6, but it is in PLAYING instead of the NULL state.
You need to explicitly set elements to the NULL state before
dropping the final reference, to allow them to clean up.
This problem may also be caused by a refcounting bug in the
application or some element.

Followed by:
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 2 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 2 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Error generated. /media/jerry/Hitachi/L4T/T186/r32.3.1/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:642 Failed to create CaptureSession
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 10000, max 200000000;

GST_ARGUS: 3840 x 2176 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 1000000, max 32000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3864 H = 2180
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: PowerService: requested_clock_Hz=110558704
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
GST_ARGUS:
PowerServiceHwVic::cleanupResources
QThread: Destroyed while thread is still running

Thanks,
Terry

Hi,
The must-have patch for r32.3.1 is
R32.3.1 tx2-4g segmentation fault in gstreamer app - #3 by DaneLLL

Please apply it and try again. We also suggest upgrade to r32.4.4 or r32.5.1.

@DaneLLL patch was in and I am still having the problem with EOS hanging and setting the state to NULL hanging on my system

I only have a camera driver for r32.3.1 so changing OS’s is not possible.

HELP
Terry

@DaneLLL

So I put in a problem report in bugzilla,

Please forward to your developers.

I can gather information for bugzilla, I can gather information for Nvidia if I was asked.

Terry

Terry

Hi,
We have shared a solution in
BUG: nvarguscamerasrc Segmentation fault - #11 by ShaneCCC

In our verification we don’t observe the issue. Do you observe it on TX2 4GB developer kit? Or your own custom board? Are you able to re-flash to a clean system and try?

Already am using the latest and greatest nvarguscamerasrc, because I was told to change a timeout in it.

Problems are seen on a tx2-8g with base OS.

The problem is because of my pipeline, I can write pipelines that work all the time, only the pipeline I need is having problems.

Can I get the source code for the things the gstreamer people were seeing possible lock problems in?

Thanks,
Terry

Assuming you’ve tried this as you’re pausing pipeline with nvarguscamerasrc, does the pipeline you need is the one going nvarguscamerasrc → CPU memory (for gkdpixbufoverlay) → NVMM I420 for nvjpegenc, as mentioned above ?

when I run the camera, it is displaying on the mipi dsi display, it is generating a pre-buffer of 5 seconds long so that if a violation is determined a video that contains the pre-buffer and 5 seconds of video after the violation, a number of jpgs might be generated during the 5 seconds of video.

The problem is that setting the state to NULL or sending the EOS to correctly clean up the pipeline does not work.

Terry

Not sure I correctly understand your case, sorry, but you didn’t really answer to my first question:
Did you try InfiniteTimeout for Argus daemon ?
Can you also try removing any CPU side plugin from this subpipeline (gdkpixbufoverlay, emboverlay…) ? (Just for figuring out the root cause)
And only set even values to nvoverlaysink’s overlay-y property ?

I change the base nvarguscamerasrc timeout from the default to a larger number, I did not do the infinity timeout.

I don’t understand how that would allow the state to be set to NULL.

I can remove things from my pipeline and things will work, the problem I have is that this is the pipeline I need, and I can’t do the gstreamer correct cleanup when terminating my pipeline.

I gave a pointer to the gstreamer problem, the gsteamer people said it looked like a lock problem, so I am looking to find if there is something I can fix to make the cleanup correct.

I can just terminate my pipeline and restart it and is close to 100% good, but it still can hang when bringing up the pipeline.

Terry

You may try this.

Indeed, I can stop this pipeline (derived from yours) without error:

gst-launch-1.0 -ev nvarguscamerasrc name=camera \
   ! 'video/x-raw(memory:NVMM),format=NV12,width=3264,height=2464,framerate=21/1,pixel-aspect-ratio=1/1' \
   ! queue max-size-buffers=1 leaky=2 name=q1 \
   ! tee name=t  \
   t. ! queue name=q3 \
     ! nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1012 right=2451 \
     ! 'video/x-raw(memory:NVMM),width=1440,height=1140' \
     ! nvvidconv \
     ! 'video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1' \
     ! nvoverlaysink name=displaySink sync=false overlay-x=0  overlay-y=360 overlay-w=480 overlay-h=380 \
   t. ! queue name=q4 \
      ! valve name=stp \
      ! nvvidconv name=stpconv \
      ! video/x-raw,format=I420 \
      ! gdkpixbufoverlay name=reticalpic location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=1682 offset-y=840  \
      ! nvvidconv \
      ! 'video/x-raw(memory:NVMM),format=I420,width=3264,height=2464,pixel-aspect-ratio=1/1' \
      ! queue ! nvjpegenc \
      ! image/jpeg,framerate=21/1 \
      ! multifilesink name=stalkerPicture location=./stalkerPic_%08d.jpg max-files=1 \
   t. ! queue name=q5 \
     ! nvvidconv name=b \
     ! video/x-raw, width=1920, height=1080 \
     ! gdkpixbufoverlay name=reticalvid location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=710 offset-y=390 \
     ! queue \
     ! nvvidconv \
     ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' \
     ! nvv4l2h264enc  maxperf-enable=1 bitrate=8000000 \
     ! video/x-h264, stream-format=byte-stream \
     ! h264parse \
     ! valve name=stalkercreatevideo \
     ! splitmuxsink name=stalkerVideoSink async-handling=true location=/dev/null 

added the sync and put the set_state back in and my set_state is still hanging

Thanks,
Terry

Would you try this (just adapt to your camera resolution). Works fine for me:

#!/usr/bin/env python

import signal
import sys
import time
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject

# Signal handler for stopping pipeline before destruction, so that Argus keeps ok.
def signal_handler(sig, frame):
    p.set_state(Gst.State.NULL)
    sys.exit(0)

# Initialize gstreamer
GObject.threads_init()
Gst.init(None)

# Define pipeline 

gst_str = "nvarguscamerasrc name=camera \
   ! video/x-raw(memory:NVMM),format=NV12,width=3264,height=2464,framerate=21/1,pixel-aspect-ratio=1/1 \
   ! queue max-size-buffers=1 leaky=2 name=q1 \
   ! tee name=t  \
   t. ! queue name=q3 \
     ! nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1012 right=2451 \
     ! video/x-raw(memory:NVMM),width=1440,height=1140 \
     ! nvvidconv \
     ! video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1 \
     ! nvoverlaysink name=displaySink overlay-x=0  overlay-y=360 overlay-w=480 overlay-h=380 \
   t. ! queue name=q4 \
      ! valve name=stp \
      ! nvvidconv name=stpconv \
      ! video/x-raw,format=I420 \
      ! gdkpixbufoverlay name=reticalpic location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=1682 offset-y=840  \
      ! nvvidconv \
      ! video/x-raw(memory:NVMM),format=I420,width=3264,height=2464,pixel-aspect-ratio=1/1 \
      ! queue ! nvjpegenc \
      ! image/jpeg,framerate=21/1 \
      ! multifilesink name=stalkerPicture location=./stalkerPic_%08d.jpg max-files=1 \
   t. ! queue name=q5 \
     ! nvvidconv name=b \
     ! video/x-raw, width=1920, height=1080 \
     ! gdkpixbufoverlay name=reticalvid location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=710 offset-y=390 \
     ! queue \
     ! nvvidconv \
     ! video/x-raw(memory:NVMM), width=1920, height=1080 \
     ! nvv4l2h264enc  maxperf-enable=1 bitrate=8000000 \
     ! video/x-h264, stream-format=byte-stream \
     ! h264parse \
     ! valve name=stalkercreatevideo \
     ! splitmuxsink name=stalkerVideoSink async-handling=true location=/dev/null async=false"



# Create the pipeline
pipeline = Gst.parse_launch (gst_str)

# Register signal handler for proper termination if receiving SIGINT such as Ctrl-C
signal.signal(signal.SIGINT, signal_handler)

# Start the pipeline
pipeline.set_state(Gst.State.READY)
pipeline.set_state(Gst.State.PAUSED)
pipeline.set_state(Gst.State.PLAYING)

# Run for 5s 
time.sleep(5)

# Done. Stop the pipeline before restarting.
pipeline.set_state(Gst.State.NULL)
#time.sleep(1)

# Because of a bug, need to reset properties of nvoverlaysink before restarting
# See: https://forums.developer.nvidia.com/t/nvoverlaysink-ignores-properties-when-pipeline-is-restarted/179379/8
overlay = pipeline.get_by_name("displaySink")
overlay.set_property("overlay-x", 0)
overlay.set_property("overlay-y", 0)
overlay.set_property("overlay-w", 0)
overlay.set_property("overlay-h", 0)

overlay.set_property("overlay-x", 0)
overlay.set_property("overlay-y", 360)
overlay.set_property("overlay-w", 480)
overlay.set_property("overlay-h", 380)


pipeline.set_state(Gst.State.READY)
pipeline.set_state(Gst.State.PAUSED)
pipeline.set_state(Gst.State.PLAYING)

# Run for 5s 
time.sleep(5)

# Done. Stop the pipeline before clean up on exit.
pipeline.set_state(Gst.State.NULL)
exit(0)

For EOS management, you may also try this:

#!/usr/bin/env python

import sys, os
import gi
gi.require_version('Gst', '1.0')
gi.require_version('Gtk', '3.0')
from gi.repository import Gst, GObject, Gtk


def verbose_deep_notify_cb(object, orig, pspec, component):
    """
    A quick attempt to mimic gst-launch verbose mode.
    """
    if pspec.value_type == Gst.Caps.__gtype__:
    	caps = orig.get_current_caps()
	if caps != None:
	   print("%s/%s/%s: caps = \"%s\"" % (object.get_name(), orig.parent.get_name(), orig.get_name(), caps.to_string()))



class GTK_Main:
    def __init__(self):
        window = Gtk.Window(Gtk.WindowType.TOPLEVEL)
        window.set_title("CamRecorder")
        window.set_default_size(100, 100)
        window.connect("destroy", Gtk.main_quit, "WM destroy")
        vbox = Gtk.VBox()
        window.add(vbox)
        hbox = Gtk.HBox()
        vbox.pack_start(hbox, False, False, 0)
        hbox.set_border_width(10)
        hbox.pack_start(Gtk.Label(), False, False, 0)
        self.button = Gtk.Button("Start")
        self.button.connect("clicked", self.start_stop)
        hbox.pack_start(self.button, False, False, 0)
        self.button2 = Gtk.Button("Quit")
        self.button2.connect("clicked", self.exit)
        hbox.pack_start(self.button2, False, False, 0)
        hbox.add(Gtk.Label())
        window.show_all()

        # Set up the gstreamer pipeline
	gst_str = "nvarguscamerasrc name=camera \
  		   ! video/x-raw(memory:NVMM),format=NV12,width=3264,height=2464,framerate=21/1,pixel-aspect-ratio=1/1 \
  		   ! queue max-size-buffers=1 leaky=2 name=q1 \
		   ! tee name=t  \
		   t. ! queue name=q3 \
		     ! nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1012 right=2451 \
		     ! video/x-raw(memory:NVMM),width=1440,height=1140 \
		     ! nvvidconv \
		     ! video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1 \
		     ! nvoverlaysink name=displaySink overlay-x=0  overlay-y=360 overlay-w=480 overlay-h=380 \
		   t. ! queue name=q4 \
		      ! valve name=stp \
		      ! nvvidconv name=stpconv \
		      ! video/x-raw,format=I420 \
		      ! gdkpixbufoverlay name=reticalpic location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=1682 offset-y=840  \
		      ! nvvidconv \
		      ! video/x-raw(memory:NVMM),format=I420,width=3264,height=2464,pixel-aspect-ratio=1/1 \
		      ! queue ! nvjpegenc \
		      ! image/jpeg,framerate=21/1 \
		      ! multifilesink name=stalkerPicture location=./stalkerPic_%08d.jpg max-files=1 \
		   t. ! queue name=q5 \
		     ! nvvidconv name=b \
		     ! video/x-raw, width=1920, height=1080 \
		     ! gdkpixbufoverlay name=reticalvid location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=710 offset-y=390 \
		     ! queue \
		     ! nvvidconv \
		     ! video/x-raw(memory:NVMM), width=1920, height=1080 \
		     ! nvv4l2h264enc  maxperf-enable=1 bitrate=8000000 \
		     ! video/x-h264, stream-format=byte-stream \
		     ! h264parse \
		     ! valve name=stalkercreatevideo \
		     ! splitmuxsink name=stalkerVideoSink async-handling=true location=/dev/null async=false"

        self.player = Gst.parse_launch (gst_str)
        self.player.connect('deep-notify', verbose_deep_notify_cb, self)
        bus = self.player.get_bus()
        bus.add_signal_watch()
        bus.enable_sync_message_emission()
        bus.connect("message", self.on_message)

    def start_stop(self, w, data=None):
        if self.button.get_label() == "Start":
            overlay = self.player.get_by_name("displaySink")
            overlay.set_property("overlay-x", 0)
            overlay.set_property("overlay-y", 0)
            overlay.set_property("overlay-w", 0)
            overlay.set_property("overlay-h", 0)
            overlay.set_property("overlay-x", 0)
            overlay.set_property("overlay-y", 360)
            overlay.set_property("overlay-w", 480)
            overlay.set_property("overlay-h", 380)

            print("Setting pipeline to READY ...")
            self.player.set_state(Gst.State.READY)
            print("Setting pipeline to PAUSED ...")
            self.player.set_state(Gst.State.PAUSED)
            print("Setting pipeline PLAYING...")
            self.player.set_state(Gst.State.PLAYING)
            self.button.set_label("Stop")
        else:
            print("Setting pipeline to NULL ...")
            self.player.set_state(Gst.State.NULL)
            self.button.set_label("Start")

    def exit(self, w):
        self.player.set_state(Gst.State.NULL)
        Gtk.main_quit()

    def on_message(self, bus, message):
        t = message.type
        if t == Gst.MessageType.EOS:
            print('Received message type EOS')
            self.player.set_state(Gst.State.NULL)
            self.button.set_label("Start")
        elif t == Gst.MessageType.ERROR:
            err, debug = message.parse_error()
            print "Error: %s" % err, debug
            self.player.set_state(Gst.State.NULL)
            self.button.set_label("Start")
	

Gst.debug_set_active(True)
Gst.debug_set_default_threshold(0)
GObject.threads_init()
Gst.init(None)
GTK_Main()
Gtk.main()

@Honey_Patouceul the python script does not work on my system, I noticed the -evvv is missing, so I removed -evvv from my system and it does not work either, so I added -evvv to the python and it gets an error, so your gst-parse-launch must be different from my c++ gst-parse-launch

Sorry I am dead in the water trying to run you python.

Terry