R32.3.1 tx2-4g gst_element_change_state hangs

trying to set my pipeline to NULL hangs.

gst_element_get_state also hangs.

Terry

Seriously, how many times do we need tell you to describe your problem clearly? We’ve been saying this comment to you for one year. Share the command or sample you are using.

If you don’t want to have another problem that cannot be resolved for one year, then please share the detail about your issue.

What more information do you need. I have a gstreamer app that is failing to terminate using the standard gstreamer procedure of setting gst_element_set_state( pipeLine, GST_STATE_NULL) and it hangs it never returns, leaving my app in a very bad state.

I tried to do a gst_element_get_state before and it hung also, what debugging can I use to help you help me.

Maybe someone with gstreamer knowledge should look at these topics, @WayneWWW does not seem to understand simple questions/problems.

Terry

I agree with Wayne.

So now we know that you have a problem with an old L4T version using some secret hardware and some secret pipeline.
Ok, but this wouldn’t help you much further.

If you’re just kidding, or like to complain, you can continue this way, but I’m afraid NVIDIA folks will get lazy (I find they’re taking care so far albeit you’re not really helping).

You may understand that there is no guessing for such systems. The possible issues space is so huge that you would have to wonder how to efficiently point it out.

I have the problem on both the Lumenera camera, and the camera that comes with the demo board.

I have posted my script multiple times, I can post it again.

I will check with management and sent you the complete git tree with all my code and all the gstream filter I have written.

But again, tell me what debugging I can add, even send new libs, I will test and get right back to you. You have a FREE developer/tester if you want to use it.

Again I am not sure if it is my problem or Nvidia’s just looking for some help/debugging aid.

Terry

@WayneWWW

New information my pipeline is showing return value of 2, the state is 3 and pending 4 when I use gst_element_get_state( pipeline, &state, &pending, “1 sec timeout”)

I added code to do a gst_element_get_state on everything in my pipeline, and they all show ret 1, state 4, pending 0

How come the pipeline shows paused even though all the elements are PLAYING.

gst_element_get_state needs a timeout to avoid the hang.

Still gst_element_set_state(pipeLine, GST_STATE_NULL) hangs and never returns.

Terry

Again, please provide more details.
What plugin state are you trying to get ?
It has no sense and no chance to help without plugin details, the pipeline it used in, if any dynamic change of properties are performed, also check if the plugin supports this and how to signal…

bbb (2.7 KB)
contains the command line that I use in my app.

This is the debugging output when I try to terminate the app. I first show the data for gst_element_get_state for the gstreamer.

I then put all the elements into NULL, and camera being set to NULL will show
CONSUMER: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
Which is GOOD.

Then it appear a timer goes off and Camera comes back up.

GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 10000, max 200000000;

GST_ARGUS: 3840 x 2176 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 1000000, max 32000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3864 H = 2180
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.

This is Bad…

setting the pipeLine to NULL will hang

Terry

“mainwindow.cpp [ 74] VideoStop() :” Calling stop
“gstinterface.cpp [1056] Stop() :” Calling quit();
“gstinterface.cpp [1068] Stop() :” mPipeline 0x55a080c060 sret 0 state 3 pending 4
“gstinterface.cpp [1072] Stop() :” Current State
“gstinterface.cpp [1081] Stop() :” camera 0x55a077a8e0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” displaySink 0x55a075dff0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkerLoc 0x55a07de0f0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stpconv 0x55a07de4e0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” reticalpic 0x55a07530d0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” emb1 0x55a0735ad0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” jpgoverlay 0x55a0734ed0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkerPicture 0x55a07261e0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” b 0x55a07decc0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” reticalvid 0x55a07244d0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” vidoverlay 0x55a07228d0 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkercreatevideo 0x55a07e7360 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” stalkerVideoSink 0x55a0808030 sret 2 state 2 Pending 4
“gstinterface.cpp [1081] Stop() :” q3 0x55a07d6310 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” q4 0x55a07d6610 sret 1 state 4 Pending 0
“gstinterface.cpp [1081] Stop() :” q5 0x55a07d6910 sret 1 state 4 Pending 0
“gstinterface.cpp [1089] Stop() :” NEW State
CONSUMER: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
“gstinterface.cpp [1099] Stop() :” camera 0x55a077a8e0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” displaySink 0x55a075dff0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkerLoc 0x55a07de0f0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stpconv 0x55a07de4e0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” reticalpic 0x55a07530d0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” emb1 0x55a0735ad0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” jpgoverlay 0x55a0734ed0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkerPicture 0x55a07261e0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” b 0x55a07decc0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” reticalvid 0x55a07244d0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” vidoverlay 0x55a07228d0 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkercreatevideo 0x55a07e7360 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” stalkerVideoSink 0x55a0808030 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” q3 0x55a07d6310 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” q4 0x55a07d6610 sret 1 state 1 Pending 0
“gstinterface.cpp [1099] Stop() :” q5 0x55a07d6910 sret 1 state 1 Pending 0

GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 10000, max 200000000;

GST_ARGUS: 3840 x 2176 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 1000000, max 32000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3864 H = 2180
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
“gstinterface.cpp [1133] Stop() :” mPipeline 0x55a080c060 sret 2 state 3 pending 4
“gstinterface.cpp [1141] Stop() :” Before quit ret 256
“gstinterface.cpp [1143] Stop() :” Calling QCoreApplication::quit();
“mainwindow.cpp [ 76] VideoStop() :” return from gstInterface::Stop

@Honey_Patouceul I am just stabbing in the dark trying to determine if shutting down the app will allow it to restart 100% of the time, so I stumble on a google that said to put the camera in NULL state, so I looked at trying to put all elements in the NULL state, hoping the pipeline would end up in the NULL state.
The pipeline did not go to NULL, and setting the pipeline to NULL never returns, so I can’t kill the app. And then I could kill the app and leave the system in a good state.

I am only shooting from the hip to try to find a solution, maybe Nvidia could give some suggestions, the camera vendor suggested killing argus-daemon and reloading the camera driver before kill the app, but no luck it can still hang coming back up.

Terry

@Honey_Patouceul google shows gst_element_set_state hangs, no clear how to fix.

Terry

Hi,
Please send EoS before switching to NULL state, and unref, re-initialize the pipeline. The sample code of termination:

    // Send EoS(End of Stream)
    gst_element_send_event ((GstElement*)gst_pipeline, gst_event_new_eos ());
    // Wait for EOS message
    bus = gst_pipeline_get_bus(GST_PIPELINE(gst_pipeline));
    gst_bus_poll(bus, GST_MESSAGE_EOS, GST_CLOCK_TIME_NONE);
    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);

    gst_object_unref(GST_OBJECT(gst_pipeline));

It helps that you provide the pipeline.
I’m unable to fully try because there are custom plugins such as stalkertakepicture that I don’t have, but I see at least 2 problems with it:

  1. The crop parameters and output are not correct (reread this). So change the crop parameters so that
    [EDIT: needs to be corrected by -1 wrt to output resolution
    right = left + 480 -1 and bottom = top + 380 -1]:
nvvidconv name=stalkerLoc flip-method=2 top=520 bottom899 left=1212 right=1691 \
     ! 'video/x-raw(memory:NVMM),width=480,height=380'

or, if your ROI is 1440x1140 and just want to rescale for display:

nvvidconv name=stalkerLoc flip-method=2 top=520 bottom=1659 left=1212 right=2651 \
     ! 'video/x-raw(memory:NVMM),width=1440,height=1140' \
     ! nvvidconv \
     ! 'video/x-raw(memory:NVMM),width=480,height=380,pixel-aspect-ratio=1/1'
  1. For using nvjpegenc, you need to have I420 format in NVMM memory:
nvvidconv \
      ! 'video/x-raw(memory:NVMM),format=I420' \
      ! nvjpegenc 

make stalkerTakePicture and stalkerCreateVideo valve, instead

Also the display is working and the jpg is being created correctly.

Will look at this after I get the app to close down correctly and restart correctly.

Thanks,
Terry

@DaneLLL
Thanks for the code snippet. I put it in and my app hangs on the gst_bus_poll, how do I debug that please

remember the pipeline is in the PAUSED state, and the pending is PLAYING.

I have put all element in the NULL state, but it did not change the pipeline state.

So my pipeline is not behaving correctly, so what do I need to look at next. Because the set_state to NULL hangs, I commented it out. looks like gst_bus_poll is hanging also,

Terry

Ok, I’ll try that, but I’ll be away from any Jetson till next week.

Maybe it can work…however for a solution that could work on various recent L4T releases, I’d suggest to be sure.

Be also aware that stopping a wrong pipeline may also trigger issues. Maybe it would be easier to stop a correctly working pipeline…some issues may only happen at close time.

I also notice from your logs that one plugin, stalkercreatevideo, returns different code and status when you’re trying to stop:

gstinterface.cpp [1081] Stop() :” stalkerVideoSink 0x55a0808030 sret 2 state 2 Pending 4

Sorry I don’t know the code and enums for your logs, but this discrepancy would make wonder if it fails without this plugin. Does using valve instead show similar error ?

I’ve further looked into your issue.

Further than the above advice:

Not sure about the odds that a fix for R32.3.1 can be soon provided, but this will hopefully help next releases to be fixed.

@DaneLLL Any feed back about my sending EOS hanging waiting for it to return?

EOS does not terminate my stream, and setting it to NULL does not terminate the stream.

So I kill the app and get a bunch of these messages.

(VideoSystem:10224): GStreamer-CRITICAL **: 15:49:55.237:
Trying to dispose element capsfilter6, but it is in PLAYING instead of the NULL state.
You need to explicitly set elements to the NULL state before
dropping the final reference, to allow them to clean up.
This problem may also be caused by a refcounting bug in the
application or some element.

Followed by:
(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 2 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 2 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Error generated. /media/jerry/Hitachi/L4T/T186/r32.3.1/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:642 Failed to create CaptureSession
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 10000, max 200000000;

GST_ARGUS: 3840 x 2176 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 3981.070801; Exposure Range min 1000000, max 32000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 3864 H = 2180
seconds to Run = 0
Frame Rate = 59.999999
GST_ARGUS: PowerService: requested_clock_Hz=110558704
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
GST_ARGUS:
PowerServiceHwVic::cleanupResources
QThread: Destroyed while thread is still running

Thanks,
Terry

Hi,
The must-have patch for r32.3.1 is
R32.3.1 tx2-4g segmentation fault in gstreamer app - #3 by DaneLLL

Please apply it and try again. We also suggest upgrade to r32.4.4 or r32.5.1.

@DaneLLL patch was in and I am still having the problem with EOS hanging and setting the state to NULL hanging on my system

I only have a camera driver for r32.3.1 so changing OS’s is not possible.

HELP
Terry

@DaneLLL

So I put in a problem report in bugzilla,

Please forward to your developers.

I can gather information for bugzilla, I can gather information for Nvidia if I was asked.

Terry

Terry