Errors starting 7 nvarguscamersrc pipeline in R32.7.1(Jetpack 4.6.1)

shassan1 · August 11, 2023, 4:10am

Hello team,

We are trying to stream 7 cameras using deepstream pipelines.

Hardware being used includes:-

Jetson AGX Xavier with jetpack4.6.1 [L4T 32.7.1]
DESIGNCORE® NVIDIA® JETSON AGX ORIN FPD-LINK™ III INTERFACE CARD
DESIGNCORE® D3RCM-IMX390-953 RUGGED CAMERA MODULE

Errors encounter

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…

Pipeline Command being used

gst-launch-1.0 -e
nvarguscamerasrc sensor-id=0 tnr-strength=1 tnr-mode=2 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1’ ! queue ! nvv4l2vp9enc maxperf-enable=true bitrate=2000000 ! rtpvp9pay mtu=1400 ! udpsink host=192.168.2.23 port=5000 sync=false async=false
nvarguscamerasrc sensor-id=1 tnr-strength=1 tnr-mode=2 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1’ ! queue ! nvv4l2vp9enc maxperf-enable=true bitrate=2000000 ! rtpvp9pay mtu=1400 ! udpsink host=192.168.2.23 port=5001 sync=false async=false
nvarguscamerasrc sensor-id=2 tnr-strength=1 tnr-mode=2 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1’ ! queue ! nvv4l2vp9enc maxperf-enable=true bitrate=2000000 ! rtpvp9pay mtu=1400 ! udpsink host=192.168.2.23 port=5002 sync=false async=false
nvarguscamerasrc sensor-id=3 tnr-strength=1 tnr-mode=2 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1’ ! queue ! nvv4l2vp9enc maxperf-enable=true bitrate=2000000 ! rtpvp9pay mtu=1400 ! udpsink host=192.168.2.23 port=5003 sync=false async=false
nvarguscamerasrc sensor-id=4 tnr-strength=1 tnr-mode=2 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1’ ! queue ! nvv4l2vp9enc maxperf-enable=true bitrate=2000000 ! rtpvp9pay mtu=1400 ! udpsink host=192.168.2.23 port=5004 sync=false async=false
nvarguscamerasrc sensor-id=5 tnr-strength=1 tnr-mode=2 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1’ ! queue ! nvv4l2vp9enc maxperf-enable=true bitrate=2000000 ! rtpvp9pay mtu=1400 ! udpsink host=192.168.2.23 port=5005 sync=false async=false
nvarguscamerasrc sensor-id=6 tnr-strength=1 tnr-mode=2 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1’ ! queue ! nvv4l2vp9enc maxperf-enable=true bitrate=2000000 ! rtpvp9pay mtu=1400 ! udpsink host=192.168.2.23 port=5006 sync=false async=false

Complete Logs

Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL …
NvMMLiteOpen : Block : BlockType = 9
NvMMLiteOpen : Block : BlockType = 9
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
Redistribute latency…
Redistribute latency…
Setting pipeline to PLAYING …
NvMMLiteBlockCreate : Block : BlockType = 9
NvMMLiteOpen : Block : BlockType = 9
NvMMLiteBlockCreate : Block : BlockType = 9
New clock: GstSystemClock
Redistribute latency…
NvMMLiteOpen : Block : BlockType = 9
NvMMLiteOpen : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteOpen : Block : BlockType = 9
NvMMLiteBlockCreate : Block : BlockType = 9
Redistribute latency…
NvMMLiteOpen : Block : BlockType = 9
Redistribute latency…
Redistribute latency…
Redistribute latency…
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
NvMMLiteBlockCreate : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 6
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 3
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 2
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 4
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: Producer has connected; continuing.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 5
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 1
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc6: TIMEOUT
Additional debug info:
Argus Error Status
EOS on shutdown enabled – waiting for EOS after Error
Waiting for EOS…
Got EOS from element “pipeline0”.
EOS received - stopping pipeline…
Execution ended after 0:00:08.014943001
Setting pipeline to PAUSED …
Setting pipeline to READY …
GST_ARGUS: Cleaning up
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
GST_ARGUS: Cleaning up
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Caught SIGSEGV
#0 0x0000007fa75d75bc in futex_reltimed_wait_cancelable (private=, reltime=0x5500000000, expected=0, futex_word=0x7f80006a38)
#1 0x0000007fa75d75bc in __pthread_cond_wait_common (abstime=0x7ff4275910, mutex=0x558125ad40, cond=0x7f80006a10) at pthread_cond_wait.c:533
#2 0x0000007fa75d75bc in __pthread_cond_timedwait (cond=0x7f80006a10, mutex=0x558125ad40, abstime=0x7ff4275910) at pthread_cond_wait.c:667
#3 0x0000007fa6a9fab4 in NvOsConditionWaitTimeout ()
#4 0x0000007fa6b3406c in nvcamerautils::ConditionVariable::wait(nvcamerautils::Mutex const&, nvcamerautils::UnscaledTimeout const&) const ()
#5 0x0000007fa6f5a2d4 in ()
#6 0x0000007fa6f5cac0 in ()
#7 0x0000007fa6ee3464 in ()
#8 0x0000007fa78b5418 in ()
#9 0x0000007fa6e36ab0 in () at /usr/lib/aarch64-linux-gnu/libgstbase-1.0.so.0
#10 0x0000007fa788f750 in _gst_debug_min ()
Spinning. Please run ‘gdb gst-launch-1.0 11779’ to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Segmentation fault (core dumped)

Potential Solution

Same type of error has been reported and solved in the following post for R32.5.0

I need same solution for R32.7.1.
Thanks in advance
-Shams

ShaneCCC · August 11, 2023, 8:29am

Did you try boost the clocks and run infinite timeout daemon?

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon


sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

shassan1 · August 13, 2023, 10:41pm

Yes, but the problem is still there

Logs after implementing above steps

Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Opening in BLOCKING MODE
Pipeline is live and does not need PREROLL …
Redistribute latency…
NvMMLiteOpen : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteOpen : Block : BlockType = 9
NvMMLiteOpen : Block : BlockType = 9
NvMMLiteOpen : Block : BlockType = 9
NvMMLiteOpen : Block : BlockType = 9
Redistribute latency…
NvMMLiteOpen : Block : BlockType = 9
Redistribute latency…
Redistribute latency…
Redistribute latency…
Redistribute latency…
Redistribute latency…
Setting pipeline to PLAYING …
NvMMLiteBlockCreate : Block : BlockType = 9
New clock: GstSystemClock
NvMMLiteOpen : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
===== NVMEDIA: NVENC =====
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
NvMMLiteBlockCreate : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 9
GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 6
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 5
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 3
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Available Sensor modes :
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Available Sensor modes :
GST_ARGUS: Running with following settings:
Camera index = 2
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

CONSUMER: Producer has connected; continuing.
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

CONSUMER: Producer has connected; continuing.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 1
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Available Sensor modes :
CONSUMER: Producer has connected; continuing.
GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 31.622776; Exposure Range min 30000, max 33333000;

GST_ARGUS: 1936 x 1096 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 2.630268, max 15.848933; Exposure Range min 2000000, max 33000000;

GST_ARGUS: Running with following settings:
Camera index = 4
Camera mode = 3
Output Stream W = 1936 H = 1096
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc6: TIMEOUT
Additional debug info:
Argus Error Status
EOS on shutdown enabled – waiting for EOS after Error
Waiting for EOS…
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD… Exiting…
CONSUMER: Done Success
Got EOS from element “pipeline0”.
EOS received - stopping pipeline…
Execution ended after 0:00:16.027691391
Setting pipeline to PAUSED …
Setting pipeline to READY …
GST_ARGUS: Cleaning up
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
Setting pipeline to NULL …
Freeing pipeline …

ShaneCCC · August 14, 2023, 3:08am

Could you try manually launch the camera one by one.

shassan1 · August 14, 2023, 4:04am

I am able to launch 3 cameras individually without any issue, upon launching 4th camera, one camera from first three goes down but can be relaunched successfully. Same goes with 5th, 6th and 7th camera, launching one of them takes down stream of anyone launched before, but can be relaunched, in this way i am able to stream all 7 cameras by launching/relaunching them individually.
Then i tried to launch them in different combination and results of this step is not constant, sometimes 4 cameras streams are launch without any error and sometimes even launching 2 cameras gives error.

ShaneCCC · August 14, 2023, 6:59am

Did you verify by v4l2-ctl to confirm?

v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video*

shassan1 · August 14, 2023, 10:08pm

I haven’t used this command before and don’t know what output to aspect but here are the logs that i get after executing the command.

unknown arguments: /dev/video1 /dev/video2 /dev/video3 /dev/video4 /dev/video5 /dev/video6

General/Common options:
–all display all information available
-C, --get-ctrl=[,…]
get the value of the controls [VIDIOC_G_EXT_CTRLS]
-c, --set-ctrl==[,=…]
set the value of the controls [VIDIOC_S_EXT_CTRLS]
-D, --info show driver info [VIDIOC_QUERYCAP]
-d, --device= use device instead of /dev/video0
if starts with a digit, then /dev/video is used
-e, --out-device= use device for output streams instead of the
default device as set with --device
if starts with a digit, then /dev/video is used
-h, --help display this help message
–help-all all options
–help-io input/output options
–help-misc miscellaneous options
–help-overlay overlay format options
–help-sdr SDR format options
–help-selection crop/selection options
–help-stds standards and other video timings options
–help-streaming streaming options
–help-tuner tuner/modulator options
–help-vbi VBI format options
–help-vidcap video capture format options
–help-vidout vidout output format options
–help-edid edid handling options
-k, --concise be more concise if possible.
-l, --list-ctrls display all controls and their values [VIDIOC_QUERYCTRL]
-L, --list-ctrls-menus
display all controls and their menus [VIDIOC_QUERYMENU]
-r, --subset=[,,]+
the subset of the N-dimensional array to get/set for control ,
for every dimension an (, ) tuple is given.
-w, --wrapper use the libv4l2 wrapper library.
–list-devices list all v4l devices
–log-status log the board status in the kernel log [VIDIOC_LOG_STATUS]
–get-priority query the current access priority [VIDIOC_G_PRIORITY]
–set-priority=
set the new access priority [VIDIOC_S_PRIORITY]
is 1 (background), 2 (interactive) or 3 (record)
–silent only set the result code, do not print any messages
–sleep= sleep , call QUERYCAP and close the file handle
–verbose turn on verbose ioctl status reporting

shassan1 · August 14, 2023, 10:09pm

When i list all cameras using ls -al /dev/video*, the output is
crw-rw----+ 1 root video 81, 0 Aug 14 16:04 /dev/video0
crw-rw----+ 1 root video 81, 18 Aug 14 16:04 /dev/video1
crw-rw----+ 1 root video 81, 20 Aug 14 16:04 /dev/video2
crw-rw----+ 1 root video 81, 22 Aug 14 16:04 /dev/video3
crw-rw----+ 1 root video 81, 24 Aug 14 16:04 /dev/video4
crw-rw----+ 1 root video 81, 26 Aug 14 16:04 /dev/video5
crw-rw----+ 1 root video 81, 28 Aug 14 16:04 /dev/video6

shassan1 · August 14, 2023, 11:07pm

I also tried using the command for individual camera instead of using wildcard, logs are attached below

sudo v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video0

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.05 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps

sudo v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video1

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.05 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps

sudo v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video2

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.05 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<c<<<<<<<<<<<<<<<<<<<< 30.03 fps

sudo v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video3

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.05 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps

sudo v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video4

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.05 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps

sudo v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video5

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.05 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps

sudo v4l2-ctl --set-fmt-video=width=1936,height=1096 -c bypass_mode=0 --stream-mmap -d /dev/video6

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.07 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.06 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.05 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.04 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.03 fps

ShaneCCC · August 15, 2023, 6:04am

What’s the position in your device tree?

shassan1 · August 15, 2023, 11:03pm

Regarding Device Tree, i had opened a topic with camera wander, but that didn’t solve the issue.
you can check the detailed discussion here Adding set_mode_delay_ms in the camera sensor device-tree for IMX390RCM sunex DSL239 192 HFOV (1000843-R) · Issue #31 · D3Engineering/d3-jetson-bsp · GitHub

ShaneCCC · August 16, 2023, 4:31am

It’s the D3 OV10650 sensor?
Looks like D3 have file internal nv bug for it.

shassan1 · August 16, 2023, 4:34am

The camera sensor is actually IMX390RCM

ShaneCCC · August 16, 2023, 4:36am

OK
The problem looks like position incorrect cause it.

shassan1 · August 16, 2023, 4:36am

I will be needing concise information to pinpoint whether the issue is on sensor side or NV and any heads up to tackle it.

shassan1 · August 16, 2023, 4:37am

Any idea how to correct it.

ShaneCCC · August 16, 2023, 4:40am

Dump the device tree to confirm it.

sudo dtc -I fs -O dts -o extracted_proc.dts /proc/device-tree

shassan1 · August 16, 2023, 4:44am

dtsdump.txt (208.2 KB)

ShaneCCC · August 16, 2023, 4:53am

Please check the position in the extracted_proc.dts

https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=position

shassan1 · August 16, 2023, 5:00am

extracted_proc.dts (899.9 KB)