R32.3.1 tx2-4g gst_element_change_state hangs

Again, please provide more details.
What plugin state are you trying to get ?
It has no sense and no chance to help without plugin details, the pipeline it used in, if any dynamic change of properties are performed, also check if the plugin supports this and how to signal…

bbb (2.7 KB)
contains the command line that I use in my app.

This is the debugging output when I try to terminate the app. I first show the data for gst_element_get_state for the gstreamer.

I then put all the elements into NULL, and camera being set to NULL will show
CONSUMER: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
Which is GOOD.

Then it appear a timer goes off and Camera comes back up.

GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 10000, max 200000000;

GST_ARGUS: 3840 x 2176 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 1000000, max 32000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3864 H = 2180
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.

This is Bad…

setting the pipeLine to NULL will hang

Terry

“mainwindow.cpp [ 74] VideoStop() :” Calling stop
“gstinterface.cpp [1056] Stop() :” Calling quit();
“gstinterface.cpp [1068] Stop() :” mPipeline 0x55a080c060 sret 0 state 3 pending 4
“gstinterface.cpp [1072] Stop() :” Current State
“gstinterface.cpp [1081] Stop() :” camera 0x55a077a8e0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” displaySink 0x55a075dff0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkerLoc 0x55a07de0f0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stpconv 0x55a07de4e0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” reticalpic 0x55a07530d0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” emb1 0x55a0735ad0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” jpgoverlay 0x55a0734ed0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkerPicture 0x55a07261e0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” b 0x55a07decc0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” reticalvid 0x55a07244d0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” vidoverlay 0x55a07228d0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkercreatevideo 0x55a07e7360 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkerVideoSink 0x55a0808030 sret 2 state 2 Pending 4
“gstinterface.cpp [1081] Stop() :” q3 0x55a07d6310 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” q4 0x55a07d6610 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” q5 0x55a07d6910 sret 1 state 4 Pending 0
“gstinterface.cpp [1089] Stop() :” NEW State
CONSUMER: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
“gstinterface.cpp [1099] Stop() :” camera 0x55a077a8e0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” displaySink 0x55a075dff0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkerLoc 0x55a07de0f0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stpconv 0x55a07de4e0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” reticalpic 0x55a07530d0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” emb1 0x55a0735ad0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” jpgoverlay 0x55a0734ed0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkerPicture 0x55a07261e0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” b 0x55a07decc0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” reticalvid 0x55a07244d0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” vidoverlay 0x55a07228d0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkercreatevideo 0x55a07e7360 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkerVideoSink 0x55a0808030 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” q3 0x55a07d6310 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” q4 0x55a07d6610 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” q5 0x55a07d6910 sret 1 state 1 Pending 0

GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 10000, max 200000000;

GST_ARGUS: 3840 x 2176 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 1000000, max 32000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3864 H = 2180
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
“gstinterface.cpp [1133] Stop() :” mPipeline 0x55a080c060 sret 2 state 3 pending 4
“gstinterface.cpp [1141] Stop() :” Before quit ret 256
“gstinterface.cpp [1143] Stop() :” Calling QCoreApplication::quit();
“mainwindow.cpp [ 76] VideoStop() :” return from gstInterface::Stop

@Honey_Patouceul I am just stabbing in the dark trying to determine if shutting down the app will allow it to restart 100% of the time, so I stumble on a google that said to put the camera in NULL state, so I looked at trying to put all elements in the NULL state, hoping the pipeline would end up in the NULL state.
The pipeline did not go to NULL, and setting the pipeline to NULL never returns, so I can’t kill the app. And then I could kill the app and leave the system in a good state.

I am only shooting from the hip to try to find a solution, maybe Nvidia could give some suggestions, the camera vendor suggested killing argus-daemon and reloading the camera driver before kill the app, but no luck it can still hang coming back up.

Terry

@Honey_Patouceul google shows gst_element_set_state hangs, no clear how to fix.

Terry

Hi,
Please send EoS before switching to NULL state, and unref, re-initialize the pipeline. The sample code of termination:

    // Send EoS(End of Stream)
    gst_element_send_event ((GstElement*)gst_pipeline, gst_event_new_eos ());
    // Wait for EOS message
    bus = gst_pipeline_get_bus(GST_PIPELINE(gst_pipeline));
    gst_bus_poll(bus, GST_MESSAGE_EOS, GST_CLOCK_TIME_NONE);
    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);

    gst_object_unref(GST_OBJECT(gst_pipeline));

It helps that you provide the pipeline.
I’m unable to fully try because there are custom plugins such as stalkertakepicture that I don’t have, but I see at least 2 problems with it:

  1. The crop parameters and output are not correct (reread this). So change the crop parameters so that
    [EDIT: needs to be corrected by -1 wrt to output resolution
    right = left + 480 -1 and bottom = top + 380 -1]:
nvvidconv name=stalkerLoc flip-method=2 top=520 bottom899 left=1212 right=1691 \
     ! 'video/x-raw(memory:NVMM),width=480,height=380'

or, if your ROI is 1440x1140 and just want to rescale for display:

nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1212 right=2651 \
     ! 'video/x-raw(memory:NVMM),width=1440,height=1140' \
     ! nvvidconv \
     ! 'video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1'
  1. For using nvjpegenc, you need to have I420 format in NVMM memory:
nvvidconv \
      ! 'video/x-raw(memory:NVMM),format=I420' \
      ! nvjpegenc 

make stalkerTakePicture and stalkerCreateVideo valve, instead

Also the display is working and the jpg is being created correctly.

Will look at this after I get the app to close down correctly and restart correctly.

Thanks,
Terry

@DaneLLL
Thanks for the code snippet. I put it in and my app hangs on the gst_bus_poll, how do I debug that please

remember the pipeline is in the PAUSED state, and the pending is PLAYING.

I have put all element in the NULL state, but it did not change the pipeline state.

So my pipeline is not behaving correctly, so what do I need to look at next. Because the set_state to NULL hangs, I commented it out. looks like gst_bus_poll is hanging also,

Terry

Ok, I’ll try that, but I’ll be away from any Jetson till next week.

Maybe it can work…however for a solution that could work on various recent L4T releases, I’d suggest to be sure.

Be also aware that stopping a wrong pipeline may also trigger issues. Maybe it would be easier to stop a correctly working pipeline…some issues may only happen at close time.

I also notice from your logs that one plugin, stalkercreatevideo, returns different code and status when you’re trying to stop:

gstinterface.cpp [1081] Stop() :” stalkerVideoSink 0x55a0808030 sret 2 state 2 Pending 4

Sorry I don’t know the code and enums for your logs, but this discrepancy would make wonder if it fails without this plugin. Does using valve instead show similar error ?

I’ve further looked into your issue.

Further than the above advice:

Not sure about the odds that a fix for R32.3.1 can be soon provided, but this will hopefully help next releases to be fixed.

@DaneLLL Any feed back about my sending EOS hanging waiting for it to return?

EOS does not terminate my stream, and setting it to NULL does not terminate the stream.

So I kill the app and get a bunch of these messages.

(VideoSystem:10224): GStreamer-CRITICAL **: 15:49:55.237:
Trying to dispose element capsfilter6, but it is in PLAYING instead of the NULL state.
You need to explicitly set elements to the NULL state before
dropping the final reference, to allow them to clean up.
This problem may also be caused by a refcounting bug in the
application or some element.

Followed by:
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 2 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 2 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Error generated. /media/jerry/Hitachi/L4T/T186/r32.3.1/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:642 Failed to create CaptureSession
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 10000, max 200000000;

GST_ARGUS: 3840 x 2176 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 1000000, max 32000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3864 H = 2180
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: PowerService: requested_clock_Hz=110558704
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
GST_ARGUS:
PowerServiceHwVic::cleanupResources
QThread: Destroyed while thread is still running

Thanks,
Terry

Hi,
The must-have patch for r32.3.1 is
R32.3.1 tx2-4g segmentation fault in gstreamer app - #3 by DaneLLL

Please apply it and try again. We also suggest upgrade to r32.4.4 or r32.5.1.

@DaneLLL patch was in and I am still having the problem with EOS hanging and setting the state to NULL hanging on my system

I only have a camera driver for r32.3.1 so changing OS’s is not possible.

HELP
Terry

@DaneLLL

So I put in a problem report in bugzilla,

Please forward to your developers.

I can gather information for bugzilla, I can gather information for Nvidia if I was asked.

Terry

Terry

Hi,
We have shared a solution in
BUG: nvarguscamerasrc Segmentation fault - #11 by ShaneCCC

In our verification we don’t observe the issue. Do you observe it on TX2 4GB developer kit? Or your own custom board? Are you able to re-flash to a clean system and try?

Already am using the latest and greatest nvarguscamerasrc, because I was told to change a timeout in it.

Problems are seen on a tx2-8g with base OS.

The problem is because of my pipeline, I can write pipelines that work all the time, only the pipeline I need is having problems.

Can I get the source code for the things the gstreamer people were seeing possible lock problems in?

Thanks,
Terry

Assuming you’ve tried this as you’re pausing pipeline with nvarguscamerasrc, does the pipeline you need is the one going nvarguscamerasrc → CPU memory (for gkdpixbufoverlay) → NVMM I420 for nvjpegenc, as mentioned above ?

when I run the camera, it is displaying on the mipi dsi display, it is generating a pre-buffer of 5 seconds long so that if a violation is determined a video that contains the pre-buffer and 5 seconds of video after the violation, a number of jpgs might be generated during the 5 seconds of video.

The problem is that setting the state to NULL or sending the EOS to correctly clean up the pipeline does not work.

Terry

Not sure I correctly understand your case, sorry, but you didn’t really answer to my first question:
Did you try InfiniteTimeout for Argus daemon ?
Can you also try removing any CPU side plugin from this subpipeline (gdkpixbufoverlay, emboverlay…) ? (Just for figuring out the root cause)
And only set even values to nvoverlaysink’s overlay-y property ?

I change the base nvarguscamerasrc timeout from the default to a larger number, I did not do the infinity timeout.

I don’t understand how that would allow the state to be set to NULL.

I can remove things from my pipeline and things will work, the problem I have is that this is the pipeline I need, and I can’t do the gstreamer correct cleanup when terminating my pipeline.

I gave a pointer to the gstreamer problem, the gsteamer people said it looked like a lock problem, so I am looking to find if there is something I can fix to make the cleanup correct.

I can just terminate my pipeline and restart it and is close to 100% good, but it still can hang when bringing up the pipeline.

Terry