R32.3.1 tx2-4g gstreamer app fails to start

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

(Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
(Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
(Argus) Error FileOperationFailed: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
(Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
(Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
(Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:526 Failed to create CameraProvider
ERROR: from element /GstPipeline:pipeline0/GstSplitMuxSink:stalkerVideoSink/GstMP4Mux:muxer: Downstream is not seekable - will not be able to create a playable file
Additional debug info:
gstqtmux.c(2780): gst_qt_mux_start_file (): /GstPipeline:pipeline0/GstSplitMuxSink:stalkerVideoSink/GstMP4Mux:muxer

What do these messages really mean?

This happens every other time I bring up my gstreamer app. What can I do to add debugging information to determine

Terry

Also I get these messages in /var/log/syslog when I terminate my gstreamer app, so How do I terminate my gstreamer app to not get these messages.

Oct 16 14:45:49 BaseSystem_0_2 nvargus-daemon[24173]: === VideoSystem[6668]: Connection closed (7F801891D0)=== VideoSystem[6668]: WARNING: CameraProvider was not destroyed before client connection terminated.=== VideoSystem[6668]: The client may have abnormally terminated. Destroying CameraProvider…=== VideoSystem[6668]: CameraProvider destroyed (0x7f79b99d70)=== VideoSystem[6668]: WARNING: Cleaning up 1 outstanding requests…=== VideoSystem[6668]: WARNING: Cleaning up 1 outstanding streams…SCF: Error InvalidState: 5 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 305)
Oct 16 14:45:49 BaseSystem_0_2 nvargus-daemon[24173]: SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
Oct 16 14:45:49 BaseSystem_0_2 nvargus-daemon[24173]: === VideoSystem[6668]: WARNING: Cleaning up 1 outstanding stream settings…=== VideoSystem[6668]: WARNING: Cleaning up 1 outstanding sessions…(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/gl/EGLStreamProducer.cpp:497) (in Mutex.cpp, function lock(), line 79)
Oct 16 14:45:49 BaseSystem_0_2 nvargus-daemon[24173]: SCF: Error BadParameter: Buffer is not pending (in src/services/gl/EGLStreamProducer.cpp, function presentBufferInternal(), line 501)
Oct 16 14:45:49 BaseSystem_0_2 nvargus-daemon[24173]: (NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
Oct 16 14:45:49 BaseSystem_0_2 nvargus-daemon[24173]: SCF: Error BadParameter: (propagating from src/services/gl/EGLStreamProducer.cpp, function presentBuffer(), line 486)
Oct 16 14:45:49 BaseSystem_0_2 nvargus-daemon[24173]: SCF: Error BadParameter: (propagating from src/components/CaptureContainerIm

5

Hi,
Looks like you use a Bayer sensor and it cannot be launched through Argus. Do you use the camera module from our partner? Please share information about the camera you use.

6

Again this happens every other time I bring up my app. The first and the third time the app runs without any of these messages. Also the pile of messages at the end are when I take down my app or when I take down nvgstcapture.

It looks like nvidia has broken termination in the streams, how and when do you plan on fixing it. nvargus_daemon is broken when I take down the gstreamer app and produces the messages when I bring the app up. IF I WAIT it looks like nvargus_daemon restarts and my I do not see the messages.

What more debugging info do you need, do a sudo tail -f /var/log/syslog and run nvgstcapture, do a control C, you should see the second set of messages, quickly re=run the nvgstcature and you should see the first set of messages.

Terry

7

Hi,
Do you observe it with the default camera ov5693? We have performed SQA tests in each release and the result is good. Wonder if the issue is specific to your cameras. Might be an issue in sensor driver.

9

I asked the vendor to look into it, and this is the response.

Here is a response from our developer. It seems he was able to reproduce the issue with Nvidia’s default camera. He does not expect that this error has anything to do with what we are supplying.

I was able to reproduce the issue with the customer provided script on a TX2 dev board. Here’s the interesting bit; the very same issue is observed even with Nvidia’s default camera ov5693 for TX2 when I ran the script with height and width modified to ov5693’s height (2592) and width (1944), respectively.

Brendan Lelieveld-Amiro | Sales Engineer | Teledyne Lumenera

I gave them this script to reproduce the problem.

this script gives the messages

gst-launch-1.0 -vvvv nvarguscamerasrc
! ‘video/x-raw(memory:NVMM),width=3864,height=2180’
! queue max-size-buffers=1 leaky=2 name=q1
! tee name=t
t. ! queue name=q3
! nvvidconv name=stalkerLoc flip-method=2 left=1692 right=2172 top=900 bottom=1280
! ‘video/x-raw(memory:NVMM),width=480,height=380’
! nvoverlaysink name=displaySink async=false sync=false overlay-x=0 overlay-y=361 overlay-w=480 overlay-h=380
t. ! queue name=q4
! valve drop=true
! nvvidconv
! clockoverlay time-format="%D %H:%M:%S" halignment=right
! nvjpegenc ! multifilesink name=stalkerPicture async=false location=/mnt/sdcard/pic1.jpg
t. ! queue name=q5
! nvvidconv name=b ! ‘video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080’
! nvivafilter cuda-process=true customer-lib-name=“libnvsample_cudaprocess.so”
! ‘video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080, format=NV12’
! nvv4l2h264enc maxperf-enable=1 bitrate=8000000
! ‘video/x-h264, stream-format=(string)byte-stream’
! h264parse
! valve drop=true
! splitmuxsink name=stalkerVideoSink async-handling=true location=/tmp/tmpFile

Some of the /var/log/syslog messages are:

Oct 29 13:03:49 BaseSystem_0_2 nvargus-daemon[2789]: === gst-launch-1.0[15621]: CameraProvider initialized (0x7f95bb7cf0)CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
Oct 29 13:03:57 BaseSystem_0_2 nvargus-daemon[2789]: === gst-launch-1.0[15621]: Connection closed (7F9B0C81D0)=== gst-launch-1.0[15621]: WARNING: CameraProvider was not destroyed before client connection terminated.=== gst-launch-1.0[15621]: The client may have abnormally terminated. Destroying CameraProvider…=== gst-launch-1.0[15621]: CameraProvider destroyed (0x7f95bb7cf0)=== gst-launch-1.0[15621]: WARNING: Cleaning up 1 outstanding requests…=== gst-launch-1.0[15621]: WARNING: Cleaning up 1 outstanding streams…SCF: Error InvalidState: 4 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 305)
Oct 29 13:03:57 BaseSystem_0_2 nvargus-daemon[2789]: SCF: Error InvalidState: (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
Oct 29 13:03:57 BaseSystem_0_2 nvargus-daemon[2789]: === gst-launch-1.0[15621]: WARNING: Cleaning up 1 outstanding stream settings…=== gst-launch-1.0[15621]: WARNING: Cleaning up 1 outstanding sessions…(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/core_scf/src/services/gl/EGLStreamProducer.cpp:497) (in Mutex.cpp, function lock(), line 79)

​Terry

10

any more information on this? My manager is asking…

Terry

11

So it appears to be getting worse, sometimes my gstreamer app does not terminate, I can restart nvargus_daemon and my app terminates.

The app might not start again, restart nvargus_daemon and the app might start and run.

What type of debugging do you guys need from me, I sent you a script that shows the problem on the standard nvidia camera, and my lumenera camera has similiar problems.

Help, I am running out of time.

Is this being worked on or am I out of luck? How can I get this pushed to the next level of support?

Terry

12

Is anybody going to answer and or work with me on this problem, how do I get this pushed up to someone who can fix/debug a problem.

Terry

13

Any info, bump, is anybody looking at this?
Terry

14

Any info, bump, is anybody looking at this?
Terry

15

Any info, bump, is anybody looking at this?
Terry

16

Any info, bump, is anybody looking at this?
Terry

17

You may:

  • set a framerate that your sensor can provide for your resolution
  • avoid valve (may not be able to deal with NVMM memory and more, or start with drop=false and see later what happens when dynamically set to true).

For reference, this seems to work fine on NX R32.4.4 with a RPi v2 camera:

gst-launch-1.0 -ev nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=3264,height=2464,framerate=21/1' ! queue max-size-buffers=1 leaky=2 name=q1 ! tee name=t      t. ! queue name=q3 ! nvvidconv name=stalkerLoc flip-method=2 left=1692 right=2172 top=900 bottom=1280 ! 'video/x-raw(memory:NVMM),width=480,height=380' ! nvoverlaysink name=displaySink overlay-x=0 overlay-y=361 overlay-w=480 overlay-h=380      t. ! queue name=q4 ! nvvidconv ! clockoverlay time-format="%D %H:%M:%S" halignment=right ! nvjpegenc ! multifilesink name=stalkerPicture location=./pic1.jpg    t. ! queue name=q5 ! nvvidconv name=b ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! nvivafilter cuda-process=true customer-lib-name="libnvsample_cudaprocess.so" ! 'video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080, format=NV12' ! nvv4l2h264enc maxperf-enable=1 bitrate=8000000 ! video/x-h264, stream-format=byte-stream ! h264parse ! splitmuxsink name=stalkerVideoSink async-handling=true location=./tmp.h264

Also note that it may help to provide your code in block mode ( </> icon) so that it is easy for others to try, you may get more help than providing your complex pipeline as such.

18

Only have a problem when I terminate the gstream.

Framerate is 30 fps and is fine.

Used valve instead of my custom filters to show nvidia what was happening, the script and my app give the same type of messages when they terminate.

So you did not see any messages in /var/log/syslog when you terminated the script. nvargus-daemon restarted.

Did you try to run the script multiple times, because one of the failures is that the script does not run twice quickly. on my system I kill the script, get a bunch of syslog messages, try to start the script again quickly, the script might start or might not start because nvargus_daemon might have restarted and might not have restarted, if my second running of the script fails, then I kill the script and restart the nvargus-daemon, and restart the script, 9 times out of 10 the script starts and runs.

So nvidia has fixed something in r32.4.x and need to give me a hint on how to fix r32.3.1, that we are using because our camera only has a driver for r32.3.1

Thanks,

20

Hi,
There is a prebuilt lib fixing segment fault on r32.3.1. Please check
BUG: nvarguscamerasrc Segmentation fault - #7 by ShaneCCC

Look a bit different from this one, but please give it a try.