Problems recording 10s with nvarguscamerasrc

I’m experiencing problems when trying to record a 10s capture from nvarguscamerasrc into a bgr file. I’m using the devkit and its standard OV5693 based onboard camera module.

I first made sure I had enough disk space and boosted my tx2 (R28.2.1):

df -H .
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        64G  6.4G   54G  11% /

sudo nvpmodel -m0
sudo /home/nvidia/jetson_clocks.sh

While with videotestsrc everything works fine with num-buffers option:

for curLoop in `seq 1 20`; do gst-launch-1.0 videotestsrc num-buffers=300 ! 'video/x-raw, width=1920, height=1080, format=BGR, framerate=30/1' ! queue ! filesink location=video.bgr | grep -A10 "(Argus)"; ls -l video.bgr; done 
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:40 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:41 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:41 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:41 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:41 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:41 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:41 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:41 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:42 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:42 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:42 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:42 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:42 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:42 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:42 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:43 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:43 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:43 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:43 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 12:43 video.bgr

With nvarguscamerasrc, with num-buffers option the size is ok but sometimes is stalls.
I use one extra terminal monitoring the file size with:

watch -n1 ls -l video.bgr

and another extra one for kernel messages:

dmesg --follow

Now, running the capture succeeded 5 times, but got stuck in 6th loop, so I issued a Ctrl-C in the terminal, that produced a translation fault in argus_deamon:

for curLoop in `seq 1 20`; do gst-launch-1.0 -e nvarguscamerasrc num-buffers=300 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw, format=BGRx' ! videoconvert ! 'video/x-raw, format=BGR' ! queue ! filesink location=video.bgr | grep -A10 "Argus"; ls -l video.bgr; done 
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 13:14 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 13:15 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 13:15 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 13:15 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1866240000 Feb 16 13:15 video.bgr
^C-rw-rw-r-- 1 nvidia nvidia 1181952000 Feb 16 13:15 video.bgr
(Argus) Error EndOfFile: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 214)
(Argus) Error EndOfFile: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 289)
(Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 306)
(Argus) Error EndOfFile: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 317)
(Argus) Error EndOfFile: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 149)
(Argus) Error EndOfFile:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 101)
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 101)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:358 Failed to create CameraProvider
(Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
(Argus) Error InvalidState:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 101)
^C-rw-rw-r-- 1 nvidia nvidia 0 Feb 16 13:16 video.bgr
^C-rw-rw-r-- 1 nvidia nvidia 1860019200 Feb 16 13:16 video.bgr
^C^C

Note the 7th loop produced a smaller file.

Fault details:

[45646.868814] argus_daemon[19969]: unhandled level 2 translation fault (11) at 0x00000060, esr 0x92000006
[45646.878277] pgd = ffffffc18119f000
[45646.881705] [00000060] *pgd=0000000242a12003, *pud=0000000242a12003, *pmd=0000000000000000

[45646.891569] CPU: 3 PID: 19969 Comm: argus_daemon Tainted: G           O    4.4.38-tegra #1
[45646.899847] Hardware name: quill (DT)
[45646.903526] task: ffffffc1ea90f080 ti: ffffffc117ab0000 task.ti: ffffffc117ab0000
[45646.911073] PC is at 0x7f7d0133c0
[45646.914408] LR is at 0x7f7d0133b8
[45646.917734] pc : [<0000007f7d0133c0>] lr : [<0000007f7d0133b8>] pstate: 80000000
[45646.925175] sp : 0000007f4a7fe450
[45646.928500] x29: 0000007f4a7fe9d0 x28: 0000007f4a7ff1e0 
[45646.933876] x27: 0000007f8ee87f48 x26: 000000000043b560 
[45646.939302] x25: 0000007f4a7fe938 x24: 0000000000000065 
[45646.944660] x23: 0000007f8dfde000 x22: 0000000000000000 
[45646.950031] x21: 0000000000000001 x20: 0000007f18d49a60 
[45646.955378] x19: 0000000000000001 x18: 0000000000000a03 
[45646.960722] x17: 0000007f8e9b92c0 x16: 0000007f7d982ef8 
[45646.966086] x15: 0000007f8ef27000 x14: 0000007f8e5a79f8 
[45646.971433] x13: 0000007f8e5a798c x12: 0000007f8e5a793c 
[45646.976785] x11: 0000007f8e5a78e0 x10: 0000007f8e5a7874 
[45646.982148] x9 : 0000007f8e5a781c x8 : 0000000000000018 
[45646.987500] x7 : 0000000000004e01 x6 : 0000000000000000 
[45646.992844] x5 : 0000007f8c242d10 x4 : 0000007f7da61bd8 
[45646.998203] x3 : 0000000000000000 x2 : 0000000000000018 
[45647.003551] x1 : 000000000000001b x0 : 0000000000000000 

[45647.010441] Library at 0x7f7d0133c0: 0x7f7c0d0000 /usr/lib/aarch64-linux-gnu/tegra/libnvidia-eglcore.so.28.2.1
[45647.020457] Library at 0x7f7d0133b8: 0x7f7c0d0000 /usr/lib/aarch64-linux-gnu/tegra/libnvidia-eglcore.so.28.2.1
[45647.030491] vdso base = 0x7f8ef26000

Then I’ve tried to use nvarguscamerasrc option timeout. It doesn’t stuck, but produces very random sizes:

for curLoop in `seq 1 20`; do gst-launch-1.0 -e nvarguscamerasrc timeout=10 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw, format=BGRx' ! videoconvert ! 'video/x-raw, format=BGR' ! queue ! filesink location=video.bgr | grep -A10 "Argus"; ls -l video.bgr; done 
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:17 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1144627200 Feb 16 13:17 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1847577600 Feb 16 13:18 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1847577600 Feb 16 13:18 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:18 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:18 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:19 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1847577600 Feb 16 13:19 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1013990400 Feb 16 13:19 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:19 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1847577600 Feb 16 13:20 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:20 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1007769600 Feb 16 13:20 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1847577600 Feb 16 13:20 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1847577600 Feb 16 13:21 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1026432000 Feb 16 13:21 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:21 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1847577600 Feb 16 13:21 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1841356800 Feb 16 13:21 video.bgr
-rw-rw-r-- 1 nvidia nvidia 1007769600 Feb 16 13:22 video.bgr

Am I missing something ?

Hi HoneyP,
nvarguscamerasrc is newly added in r28.2.1 and it may be with issue in certain cases.

Checking the pipeline you run:

gst-launch-1.0 -e nvarguscamerasrc num-buffers=300 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw, format=BGRx' ! videoconvert ! 'video/x-raw, format=BGR' ! queue ! filesink location=video.bgr

It takes much time in videoconvert doing BGRX -> BGR conversion, which possibly leads to race condition in nvarguscamerasrc. Verified pipelines are video preview ‘nvarguscamerasrc ! nvoverlaysink’ and video encoding ‘nvcamerasrc ! omxh264enc ! filesink’.

Is it possible to use nvcamerasrc in your case to get BGR?

Hi Honey_Patouceul,

Test your pipeline with my script, I don’t see the segment fault issue on r28.2.1/TX2.
Attached file for you reference.
**Please rename recording.sh.tar to recording.sh
recording.sh.tar (327 Bytes)

@carolyuu, @daneLLL,

Thanks for your answers. I have no problem for recording with nvcamerasrc. I was just reporting some unexpected behavior with nvarguscamerasrc as it seems to replace nvcamerasrc in future releases such as R31 on Xavier.

It doesn’t seem to be related to videoconvert taking so much time, I’ve seen similar (although ending in another lib trace) translation fault running:

for curLoop in `seq 1 20`; do gst-launch-1.0 -e nvarguscamerasrc num-buffers=300 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw, format=NV12' ! queue ! filesink location=video.nv12 ; ls -l video.nv12; done

It seems a bit random, not sure if 20 runs can trigger it with high probability.

On Xavier with R31.1, I haven’t hit this error with nvarguscamerasrc and num-buffers option. However, with timeout option I do see varying lengths (in lesser extent than on tx2, though). May you share about timeout option scheme ?

Hi HoneyP,
The property should not guarantee the same amount of capture frames in every run.

timeout             : timeout to capture in seconds (Either specify timeout or num-buffers, not both)
                      flags: readable, writable

I think for large frame size, the file size may vary in large range. If you run

gst-launch-1.0 -e nvarguscamerasrc num-buffers=300 ! 'video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1' ! nvvidconv ! 'video/x-raw, format=NV12'! queue ! filesink location=video.nv12

Although it is possible to see different file size, but should be in small range such as 1920x1080x1.5x1(or 2, 3,4)