Really Buggy Behavior with Cameras

We’ve been facing various buggy behavior with our cameras.

Release Version:
# R32 (release), REVISION: 4.2, GCID: 20074772, BOARD: t210ref, EABI: aarch64, DATE: Thu Apr 9 01:22:12 UTC 2020

Steps to Reproduce the Issue:

Run the following in the terminal:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvoverlaysink

This works about once every 4 times, and requires restarts between attempts (nothing else is changed). We’re using this camera

Could you check with below command as much times to check if can get data from sensor continuously.

v4l2-ctl --stream-mmap --set-ctrl bypass_mode=0 --stream-count=100

This command doesn’t seem to do anything, is the syntax correctly? I get the error:

v4l2-ctl: unrecognized option '--stream-mmap–set-ctrl'
Unknown argument 'bypass_mode=0'

After running v4l2-ctl -d /dev/video1 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100, I get data from one camera saying VIDIOC_REQBUFS: failed: Device or resource busy, and the other works fine.

On my second Jetson Nano, both streams magically started working against (with gstreamer and the v4l2 command).

After a reboot on the first jetson, the first camera /dev/video0 works with v4l2, but not with gst-launch.

❯ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=15/1' ! nvoverlaysink

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 215)
(Argus) Error Timeout: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 102)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:551 Failed to create CameraProvider
Got EOS from element "pipeline0".
Execution ended after 0:01:00.010640759
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Inspecting the log at /var/log/syslog gets this:

May 28 10:48:50 jetson-nano nvargus-daemon[5161]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)
May 28 10:48:50 jetson-nano nvargus-daemon[5161]: (Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
May 28 10:48:50 jetson-nano nvargus-daemon[5161]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)

Jetson two is still working fine

After switching the SD cards between the two Jetsons, the first Jetson will work /dev/video0 a single time (and never with /dev/video1, then running again it won’t work at all. Same invalid state errors in the log.

Update, the cameras seem to be corrupting the camera driver. After running the gst-launch-1.0 nvargussrc command, the Jetson crashed and rebooted. The logs are filled with invalidstate errors :(. The behavior still isn’t really consistent either.

To make this whole thread more clear, we have this configuration:

Jetson 1 SD Card A

  • Camera 1A
  • Camera 1B

Jetson 2 SD Card B

  • Camera 2A
  • Camera 2B

Camera 3A

When first testing this stuff out, none of the cameras were working. The next day, Camera 2A and B were working fine with Jetson 2, but Camera 1A and B weren’t working with Jetson 1. We then swapped the SD cards so Jetson 2 had SD Card A. Jetson 2 with SD Card A was fine with Cameras 2A and 2B. Jetson 1 with SD Card B wouldn’t work with Cameras 1A and B. Further testing showed that if we introduced Camera3A into the mix, it worked fine with Jetson 1 and SD Card B on both CSI connectors. We then connected Jetson 2 with SD card A to Camera 1A, which worked fine (odd considering it didn’t work when SD Card A was on Jetson 1), but Jetson 2 wouldn’t connect to Camera 1B. Camera 1B I think may be the one erroring everything out, as it crashed Jetson 2 with SD Card A after running

I would suggest use v4l2-ctl only to collect/narrow down the problem.

Hi @ShaneCCC, I’m seeing the exact same issue as described in this thread where gstreamer will fail to load the cameras seemingly randomly (errors attached below).

I need to use gstreamer since it is deeply integrated with the rest of my codebase. Can you suggest how I can debug the issue?

Errors:

(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:551 Failed to create CameraProvider
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:551 Failed to create CameraProvider
End of stream
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 87)
(Argus) Error InvalidState: Argus client is exiting with 4 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)

I’m running the following pipelines:

nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=354, height=240, framerate=30/1  ! nvv4l2vp8enc maxperf-enable=1 ! rtpvp8pay ssrc=5000 pt=96 ! appsink name=appsink
nvarguscamerasrc sensor-id=1 ! video/x-raw(memory:NVMM), width=354, height=240, framerate=30/1  ! nvv4l2vp8enc maxperf-enable=1 ! rtpvp8pay ssrc=5000 pt=96 ! appsink name=appsink
1 Like

Did you confirm the v4l2-ctl can capture data continuously?