R32.3.1 tx2-4g gst_element_change_state hangs

Hi,
We have shared a solution in
BUG: nvarguscamerasrc Segmentation fault - #11 by ShaneCCC

In our verification we don’t observe the issue. Do you observe it on TX2 4GB developer kit? Or your own custom board? Are you able to re-flash to a clean system and try?

Already am using the latest and greatest nvarguscamerasrc, because I was told to change a timeout in it.

Problems are seen on a tx2-8g with base OS.

The problem is because of my pipeline, I can write pipelines that work all the time, only the pipeline I need is having problems.

Can I get the source code for the things the gstreamer people were seeing possible lock problems in?

Thanks,
Terry

Assuming you’ve tried this as you’re pausing pipeline with nvarguscamerasrc, does the pipeline you need is the one going nvarguscamerasrc → CPU memory (for gkdpixbufoverlay) → NVMM I420 for nvjpegenc, as mentioned above ?

when I run the camera, it is displaying on the mipi dsi display, it is generating a pre-buffer of 5 seconds long so that if a violation is determined a video that contains the pre-buffer and 5 seconds of video after the violation, a number of jpgs might be generated during the 5 seconds of video.

The problem is that setting the state to NULL or sending the EOS to correctly clean up the pipeline does not work.

Terry

Not sure I correctly understand your case, sorry, but you didn’t really answer to my first question:
Did you try InfiniteTimeout for Argus daemon ?
Can you also try removing any CPU side plugin from this subpipeline (gdkpixbufoverlay, emboverlay…) ? (Just for figuring out the root cause)
And only set even values to nvoverlaysink’s overlay-y property ?

I change the base nvarguscamerasrc timeout from the default to a larger number, I did not do the infinity timeout.

I don’t understand how that would allow the state to be set to NULL.

I can remove things from my pipeline and things will work, the problem I have is that this is the pipeline I need, and I can’t do the gstreamer correct cleanup when terminating my pipeline.

I gave a pointer to the gstreamer problem, the gsteamer people said it looked like a lock problem, so I am looking to find if there is something I can fix to make the cleanup correct.

I can just terminate my pipeline and restart it and is close to 100% good, but it still can hang when bringing up the pipeline.

Terry

You may try this.

Indeed, I can stop this pipeline (derived from yours) without error:

gst-launch-1.0 -ev nvarguscamerasrc name=camera \
   ! 'video/x-raw(memory:NVMM),format=NV12,width=3264,height=2464,framerate=21/1,pixel-aspect-ratio=1/1' \
   ! queue max-size-buffers=1 leaky=2 name=q1 \
   ! tee name=t  \
   t. ! queue name=q3 \
     ! nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1012 right=2451 \
     ! 'video/x-raw(memory:NVMM),width=1440,height=1140' \
     ! nvvidconv \
     ! 'video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1' \
     ! nvoverlaysink name=displaySink sync=false overlay-x=0  overlay-y=360 overlay-w=480 overlay-h=380 \
   t. ! queue name=q4 \
      ! valve name=stp \
      ! nvvidconv name=stpconv \
      ! video/x-raw,format=I420 \
      ! gdkpixbufoverlay name=reticalpic location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=1682 offset-y=840  \
      ! nvvidconv \
      ! 'video/x-raw(memory:NVMM),format=I420,width=3264,height=2464,pixel-aspect-ratio=1/1' \
      ! queue ! nvjpegenc \
      ! image/jpeg,framerate=21/1 \
      ! multifilesink name=stalkerPicture location=./stalkerPic_%08d.jpg max-files=1 \
   t. ! queue name=q5 \
     ! nvvidconv name=b \
     ! video/x-raw, width=1920, height=1080 \
     ! gdkpixbufoverlay name=reticalvid location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=710 offset-y=390 \
     ! queue \
     ! nvvidconv \
     ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' \
     ! nvv4l2h264enc  maxperf-enable=1 bitrate=8000000 \
     ! video/x-h264, stream-format=byte-stream \
     ! h264parse \
     ! valve name=stalkercreatevideo \
     ! splitmuxsink name=stalkerVideoSink async-handling=true location=/dev/null 

added the sync and put the set_state back in and my set_state is still hanging

Thanks,
Terry

Would you try this (just adapt to your camera resolution). Works fine for me:

#!/usr/bin/env python

import signal
import sys
import time
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject

# Signal handler for stopping pipeline before destruction, so that Argus keeps ok.
def signal_handler(sig, frame):
    p.set_state(Gst.State.NULL)
    sys.exit(0)

# Initialize gstreamer
GObject.threads_init()
Gst.init(None)

# Define pipeline 

gst_str = "nvarguscamerasrc name=camera \
   ! video/x-raw(memory:NVMM),format=NV12,width=3264,height=2464,framerate=21/1,pixel-aspect-ratio=1/1 \
   ! queue max-size-buffers=1 leaky=2 name=q1 \
   ! tee name=t  \
   t. ! queue name=q3 \
     ! nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1012 right=2451 \
     ! video/x-raw(memory:NVMM),width=1440,height=1140 \
     ! nvvidconv \
     ! video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1 \
     ! nvoverlaysink name=displaySink overlay-x=0  overlay-y=360 overlay-w=480 overlay-h=380 \
   t. ! queue name=q4 \
      ! valve name=stp \
      ! nvvidconv name=stpconv \
      ! video/x-raw,format=I420 \
      ! gdkpixbufoverlay name=reticalpic location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=1682 offset-y=840  \
      ! nvvidconv \
      ! video/x-raw(memory:NVMM),format=I420,width=3264,height=2464,pixel-aspect-ratio=1/1 \
      ! queue ! nvjpegenc \
      ! image/jpeg,framerate=21/1 \
      ! multifilesink name=stalkerPicture location=./stalkerPic_%08d.jpg max-files=1 \
   t. ! queue name=q5 \
     ! nvvidconv name=b \
     ! video/x-raw, width=1920, height=1080 \
     ! gdkpixbufoverlay name=reticalvid location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=710 offset-y=390 \
     ! queue \
     ! nvvidconv \
     ! video/x-raw(memory:NVMM), width=1920, height=1080 \
     ! nvv4l2h264enc  maxperf-enable=1 bitrate=8000000 \
     ! video/x-h264, stream-format=byte-stream \
     ! h264parse \
     ! valve name=stalkercreatevideo \
     ! splitmuxsink name=stalkerVideoSink async-handling=true location=/dev/null async=false"



# Create the pipeline
pipeline = Gst.parse_launch (gst_str)

# Register signal handler for proper termination if receiving SIGINT such as Ctrl-C
signal.signal(signal.SIGINT, signal_handler)

# Start the pipeline
pipeline.set_state(Gst.State.READY)
pipeline.set_state(Gst.State.PAUSED)
pipeline.set_state(Gst.State.PLAYING)

# Run for 5s 
time.sleep(5)

# Done. Stop the pipeline before restarting.
pipeline.set_state(Gst.State.NULL)
#time.sleep(1)

# Because of a bug, need to reset properties of nvoverlaysink before restarting
# See: https://forums.developer.nvidia.com/t/nvoverlaysink-ignores-properties-when-pipeline-is-restarted/179379/8
overlay = pipeline.get_by_name("displaySink")
overlay.set_property("overlay-x", 0)
overlay.set_property("overlay-y", 0)
overlay.set_property("overlay-w", 0)
overlay.set_property("overlay-h", 0)

overlay.set_property("overlay-x", 0)
overlay.set_property("overlay-y", 360)
overlay.set_property("overlay-w", 480)
overlay.set_property("overlay-h", 380)


pipeline.set_state(Gst.State.READY)
pipeline.set_state(Gst.State.PAUSED)
pipeline.set_state(Gst.State.PLAYING)

# Run for 5s 
time.sleep(5)

# Done. Stop the pipeline before clean up on exit.
pipeline.set_state(Gst.State.NULL)
exit(0)

For EOS management, you may also try this:

#!/usr/bin/env python

import sys, os
import gi
gi.require_version('Gst', '1.0')
gi.require_version('Gtk', '3.0')
from gi.repository import Gst, GObject, Gtk


def verbose_deep_notify_cb(object, orig, pspec, component):
    """
    A quick attempt to mimic gst-launch verbose mode.
    """
    if pspec.value_type == Gst.Caps.__gtype__:
    	caps = orig.get_current_caps()
	if caps != None:
	   print("%s/%s/%s: caps = \"%s\"" % (object.get_name(), orig.parent.get_name(), orig.get_name(), caps.to_string()))



class GTK_Main:
    def __init__(self):
        window = Gtk.Window(Gtk.WindowType.TOPLEVEL)
        window.set_title("CamRecorder")
        window.set_default_size(100, 100)
        window.connect("destroy", Gtk.main_quit, "WM destroy")
        vbox = Gtk.VBox()
        window.add(vbox)
        hbox = Gtk.HBox()
        vbox.pack_start(hbox, False, False, 0)
        hbox.set_border_width(10)
        hbox.pack_start(Gtk.Label(), False, False, 0)
        self.button = Gtk.Button("Start")
        self.button.connect("clicked", self.start_stop)
        hbox.pack_start(self.button, False, False, 0)
        self.button2 = Gtk.Button("Quit")
        self.button2.connect("clicked", self.exit)
        hbox.pack_start(self.button2, False, False, 0)
        hbox.add(Gtk.Label())
        window.show_all()

        # Set up the gstreamer pipeline
	gst_str = "nvarguscamerasrc name=camera \
  		   ! video/x-raw(memory:NVMM),format=NV12,width=3264,height=2464,framerate=21/1,pixel-aspect-ratio=1/1 \
  		   ! queue max-size-buffers=1 leaky=2 name=q1 \
		   ! tee name=t  \
		   t. ! queue name=q3 \
		     ! nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1012 right=2451 \
		     ! video/x-raw(memory:NVMM),width=1440,height=1140 \
		     ! nvvidconv \
		     ! video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1 \
		     ! nvoverlaysink name=displaySink overlay-x=0  overlay-y=360 overlay-w=480 overlay-h=380 \
		   t. ! queue name=q4 \
		      ! valve name=stp \
		      ! nvvidconv name=stpconv \
		      ! video/x-raw,format=I420 \
		      ! gdkpixbufoverlay name=reticalpic location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=1682 offset-y=840  \
		      ! nvvidconv \
		      ! video/x-raw(memory:NVMM),format=I420,width=3264,height=2464,pixel-aspect-ratio=1/1 \
		      ! queue ! nvjpegenc \
		      ! image/jpeg,framerate=21/1 \
		      ! multifilesink name=stalkerPicture location=./stalkerPic_%08d.jpg max-files=1 \
		   t. ! queue name=q5 \
		     ! nvvidconv name=b \
		     ! video/x-raw, width=1920, height=1080 \
		     ! gdkpixbufoverlay name=reticalvid location=/usr/share/unity-control-center/icons/hicolor/48x48/devices/audio-speaker-left-side.svg offset-x=710 offset-y=390 \
		     ! queue \
		     ! nvvidconv \
		     ! video/x-raw(memory:NVMM), width=1920, height=1080 \
		     ! nvv4l2h264enc  maxperf-enable=1 bitrate=8000000 \
		     ! video/x-h264, stream-format=byte-stream \
		     ! h264parse \
		     ! valve name=stalkercreatevideo \
		     ! splitmuxsink name=stalkerVideoSink async-handling=true location=/dev/null async=false"

        self.player = Gst.parse_launch (gst_str)
        self.player.connect('deep-notify', verbose_deep_notify_cb, self)
        bus = self.player.get_bus()
        bus.add_signal_watch()
        bus.enable_sync_message_emission()
        bus.connect("message", self.on_message)

    def start_stop(self, w, data=None):
        if self.button.get_label() == "Start":
            overlay = self.player.get_by_name("displaySink")
            overlay.set_property("overlay-x", 0)
            overlay.set_property("overlay-y", 0)
            overlay.set_property("overlay-w", 0)
            overlay.set_property("overlay-h", 0)
            overlay.set_property("overlay-x", 0)
            overlay.set_property("overlay-y", 360)
            overlay.set_property("overlay-w", 480)
            overlay.set_property("overlay-h", 380)

            print("Setting pipeline to READY ...")
            self.player.set_state(Gst.State.READY)
            print("Setting pipeline to PAUSED ...")
            self.player.set_state(Gst.State.PAUSED)
            print("Setting pipeline PLAYING...")
            self.player.set_state(Gst.State.PLAYING)
            self.button.set_label("Stop")
        else:
            print("Setting pipeline to NULL ...")
            self.player.set_state(Gst.State.NULL)
            self.button.set_label("Start")

    def exit(self, w):
        self.player.set_state(Gst.State.NULL)
        Gtk.main_quit()

    def on_message(self, bus, message):
        t = message.type
        if t == Gst.MessageType.EOS:
            print('Received message type EOS')
            self.player.set_state(Gst.State.NULL)
            self.button.set_label("Start")
        elif t == Gst.MessageType.ERROR:
            err, debug = message.parse_error()
            print "Error: %s" % err, debug
            self.player.set_state(Gst.State.NULL)
            self.button.set_label("Start")
	

Gst.debug_set_active(True)
Gst.debug_set_default_threshold(0)
GObject.threads_init()
Gst.init(None)
GTK_Main()
Gtk.main()

@Honey_Patouceul the python script does not work on my system, I noticed the -evvv is missing, so I removed -evvv from my system and it does not work either, so I added -evvv to the python and it gets an error, so your gst-parse-launch must be different from my c++ gst-parse-launch

Sorry I am dead in the water trying to run you python.

Terry

@DaneLLL any more things to try, skipping the state change still can hang while trying to terminate my video app. So instead of not coming up 100% of the time, it now does not terminate 100% of the time.

What information can I gather to help you solve the problems.
Terry

There is a problem with EOS handling / restarting by splitmuxsink.
Without it, sending EOS works fine.
With it, it doesn’t work on restarting and gets stuck with second EOS.
See this topic.
You may try to find an alternative solution for this.

Hi,
We now have remote access to Jetson platforms and the devices are in r32.5.1(Jetpack 4.5.1). Will try to set up a r32.3.1 TX2 and reproduce the issue. Due to pandemic, please expect some delay.

Hi @terrysu50z
Please check if the python script works with your camera sensor:

#!/usr/bin/env python3

import gi
import time
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject, GLib

pipeline = None
bus = None
message = None

# initialize GStreamer
Gst.init(None)

for i in range(1, 5):
    print("loop =",i," ")
    # build the pipeline
    pipeline = Gst.parse_launch(
        "nvarguscamerasrc ! nvvidconv ! fakesink "
    )

    # start playing
    print("Switch to PLAYING state")
    pipeline.set_state(Gst.State.PLAYING)

    time.sleep(5)
    print("Send EoS")
    Gst.Element.send_event(pipeline, Gst.Event.new_eos())
    # wait until EOS or error
    bus = pipeline.get_bus()
    msg = bus.timed_pop_filtered(
        Gst.CLOCK_TIME_NONE, Gst.MessageType.EOS)

    # free resources
    print("Switch to NULL state")
    pipeline.set_state(Gst.State.NULL)
    time.sleep(1)

We don’t hit issue in our r32.3.1/TX2 + default camera board. Please give it a try.

@DaneLLL

your python script works perfectly on my system, which I already knew. My problem is because my script does not go into the PLAYING state, so it will not responded to any of the gst commands.

Look at Honey’s script and see if you have problems. Mess with the script and get it into the state where it is playing but not in the playing state.

Terry

Hi,
Please share detail about the use-case. Not sure why you initialize a gstreamer pipeline and don’t go to PLAYING state.

my command line is attached, and Honey had a python script that ran my system.

Clarification, I put my pipeline into PLAYING, it did not go into PLAYING, but the video displays on the screen, I can take jpgs and I can create mpgs, so the script/pipeline is not in playing but does everything I need and it is working/running, except allow me to set it to NULL, and release the pipeline cleanly.

Someone on the internet said to use omxh264enc instead of nv264enc, any truth to that?

Terry

Hi Terrysu50z,

The example I’ve given in second part of this post was only handling EOS from another plugin such as filesrc rather than properly sending EOS as does Dane’s script. You may adapt.

From what I remember trying your case, the issue was with splitmuxsink. Did you try replacing with fakesink (along with all previous advice) ?