Gst-launch call with nvarguscamerasrc crashes/reboots Jetson Nano when called multiple times

Hi,

we are using this Jetson Nano Development Kit. Connected to this Kit is a Camera with an IMX485 Sensor via CSI.

We need the System to take pictures on demand. Therefore we use gst-launch with nvarguscamerasrc at the moment. The exact command is:

gst-launch-1.0 nvarguscamerasrc num-buffers=3 sensor-id=0 awblock=true aelock=true ! 'video/x-raw(memory:NVMM),width=3840,height=2180,framerate=25/1' ! nvvidconv flip-method=3 ! nvjpegenc ! multifilesink location=./camera.jpeg

During our testing, the system crashed every once in a while, and we figured, that it is caused by the command above. For testing purposes, we created a loop script that will run the command until the system crashes. The script is the follwing:

#!/bin/bash
count=1
while true
do
gst-launch-1.0 nvarguscamerasrc num-buffers=3 sensor-id=0 awblock=true aelock=true ! 'video/x-raw(memory:NVMM),width=3840,height=2180,framerate=25/1' ! nvvidconv flip-method=3 ! nvjpegenc ! multifilesink location=./camera_$count.jpeg
let count++
#sleep 2
echo ======================
echo Count = $count
echo ======================
done

The Script will “reliably” crash the system at one point, usually after 70-120 iterations.

We already tried another Camera with an IMX485 sensor with another Driver and DTB, but we get the same result. We also tried to use “pngenc” instead of “nvjpegenc”, but it did not help either. Last but not least we tried this on Different Development Kits, just to make sure the first one isn’t broken.

Test System:

NVIDIA Jetson Nano Development Kit B01
Jetpack 4.6.1
L4T 32.7.1
The OS is running on an SSD connected via USB3

The same error does not occur when using v4l2-ctl, so i guess it is not a driver issue.

I attached some logs of the loop-script.

Any help/hints would be greatly appreciated.

Thank You
Timo

log_png2.txt (344 KB)
log_png1.txt (224 KB)
log_jpeg2.txt (148 KB)
log_jpeg1.txt (92 KB)

EDIT

After a few more runs i get the following error that got cut off in the log files i think:

GST_ARGUS: Running with following settings:
   Camera index = 0
   Camera mode  = 0
   Output Stream W = 3840 H = 2160
   seconds to Run    = 0
   Frame Rate = 33,000000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:00:02.275211690
Setting pipeline to PAUSED ...
Setting pipeline to READY ...

Why comment sleep? Does sleep help on the issue?

I run your script about 5634 times on Nano without problem.
The sensor is R-PI IMX219
nvidia@ubuntu:~$ cat /etc/nv_tegra_release

R32 (release), REVISION: 7.2, GCID: 30192233, BOARD: t210ref, EABI: aarch64, DATE: Wed Apr 20 21:34:48 UTC 2022

======================
Count = 5634
======================
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0
   Camera mode  = 0
   Output Stream W = 3264 H = 2464
   seconds to Run    = 0
   Frame Rate = 21.000000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds

@ShaneCCC thank you for your effort on testing the script on your machine. In general, it is good to hear that the process should work fine. Today I’m out of office, so I will do further testing on Monday.

Regarding the sleep command, we just tried whether it makes a difference, but it did not.

Thank you
Timo

I gathered the log of the nvargus-daemon.service:

Aug 30 15:04:29 linux nvargus-daemon[5263]: === gst-launch-1.0[18119]: CameraProvider destroyed (0x7f75a81600)=== gst-launch-1.0[18119]: Connection closed (7F78BB51D0)=== gst-launch-1.0[18119]: Connection cleaned up (7F78BB51D0)=== gst-launch-1.0[18193]: Connection established (7F78BB51D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Aug 30 15:04:29 linux nvargus-daemon[5263]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Aug 30 15:04:29 linux nvargus-daemon[5263]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Aug 30 15:04:29 linux nvargus-daemon[5263]: NvPclHwGetModuleList: No module data found
Aug 30 15:04:29 linux nvargus-daemon[5263]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Aug 30 15:04:29 linux nvargus-daemon[5263]: ---- imager: No override file found. ----
Aug 30 15:04:29 linux nvargus-daemon[5263]: LSC: LSC surface is not based on full res!
Aug 30 15:04:29 linux nvargus-daemon[5263]: ---- imager: No override file found. ----
Aug 30 15:04:29 linux nvargus-daemon[5263]: (NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
Aug 30 15:04:29 linux nvargus-daemon[5263]: (NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
Aug 30 15:04:29 linux nvargus-daemon[5263]: (NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
Aug 30 15:04:29 linux nvargus-daemon[5263]: NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
Aug 30 15:04:29 linux nvargus-daemon[5263]: NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
Aug 30 15:04:29 linux nvargus-daemon[5263]: NvPclStartPlatformDrivers: Failed to start module drivers
Aug 30 15:04:29 linux nvargus-daemon[5263]: NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
Aug 30 15:04:29 linux nvargus-daemon[5263]: NvPclOpen: PCL Open Failed. Error: 0xf
Aug 30 15:04:29 linux nvargus-daemon[5263]: SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 593)
Aug 30 15:04:29 linux nvargus-daemon[5263]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 437)
Aug 30 15:04:29 linux nvargus-daemon[5263]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 305)
Aug 30 15:04:29 linux nvargus-daemon[5263]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 471)
Aug 30 15:04:29 linux nvargus-daemon[5263]: Acquiring SCF Camera device source via index 1 has failed. === gst-launch-1.0[18193]: CameraProvider initialized (0x7f75d9eb00)LSC: LSC surface is not based on full res!
Aug 30 15:04:30 linux nvargus-daemon[5263]: === gst-launch-1.0[18193]: CameraProvider destroyed (0x7f75d9eb00)=== gst-launch-1.0[18193]: Connection closed (7F78BB51D0)=== gst-launch-1.0[18193]: Connection cleaned up (7F78BB51D0)=== gst-launch-1.0[18267]: Connection established (7F78BB51D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Aug 30 15:04:30 linux nvargus-daemon[5263]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Aug 30 15:04:30 linux nvargus-daemon[5263]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Aug 30 15:04:30 linux nvargus-daemon[5263]: NvPclHwGetModuleList: No module data found
Aug 30 15:04:31 linux nvargus-daemon[5263]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Aug 30 15:04:31 linux nvargus-daemon[5263]: ---- imager: No override file found. ----
Aug 30 15:04:31 linux nvargus-daemon[5263]: LSC: LSC surface is not based on full res!
Aug 30 15:04:31 linux nvargus-daemon[5263]: ---- imager: No override file found. ----
Aug 30 15:04:31 linux nvargus-daemon[5263]: (NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
Aug 30 15:04:31 linux nvargus-daemon[5263]: (NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
Aug 30 15:04:31 linux nvargus-daemon[5263]: (NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
Aug 30 15:04:31 linux nvargus-daemon[5263]: NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
Aug 30 15:04:31 linux nvargus-daemon[5263]: NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
Aug 30 15:04:31 linux nvargus-daemon[5263]: NvPclStartPlatformDrivers: Failed to start module drivers
Aug 30 15:04:31 linux nvargus-daemon[5263]: NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
Aug 30 15:04:31 linux nvargus-daemon[5263]: NvPclOpen: PCL Open Failed. Error: 0xf
Aug 30 15:04:31 linux nvargus-daemon[5263]: SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 593)
Aug 30 15:04:31 linux nvargus-daemon[5263]: SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 437)
Aug 30 15:04:31 linux nvargus-daemon[5263]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 305)
Aug 30 15:04:31 linux nvargus-daemon[5263]: SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 471)
Aug 30 15:04:31 linux nvargus-daemon[5263]: Acquiring SCF Camera device source via index 1 has failed. === gst-launch-1.0[18267]: CameraProvider initialized (0x7f75abb8a0)LSC: LSC surface is not based on full res!
Aug 30 15:04:36 linux nvargus-daemon[5263]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Aug 30 15:04:36 linux nvargus-daemon[5263]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc

I also stumbled across this Thread on the NVIDIA Forum, which seems to describe a similar issue. I already set “set_mode_delay_ms” in my dtb as follows (5000 just for testing):

           imx485_a@1a {
				status = "okay";
				compatible = "nvidia,eimx485";
				reg = <0x1a>;
				sensor_model = "eimx485";
				physical_w = "5.70";
				physical_h = "4.28";
				disable-framesync;
				reset-gpios = <0xcc 0x1 0x0>;
				use_decibel_gain = "true";
				use_sensor_mode_id = "true";
				linux,phandle = <0x12a>;
				phandle = <0x12a>;
				set_mode_delay_ms = "5000";

				mode0 {
					mclk_khz = "37125";
					num_lanes = [32 00];
					tegra_sinterface = "serial_a";
					phy_mode = "DPHY";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = [30 00];
					dynamic_pixel_bit_depth = "12";
					csi_pixel_bit_depth = "12";
					mode_type = "bayer";
					pixel_phase = "rggb";
					active_w = "3840";
					active_h = "2160";
					readout_orientation = [30 00];
					line_length = "988";
					inherent_gain = [31 00];
					mclk_multiplier = [31 00];
					pix_clk_hz = "950400000";
					gain_factor = "10";
					min_gain_val = [30 00];
					max_gain_val = "300";
					step_gain_val = [33 00];
					default_gain = [30 00];
					min_hdr_ratio = [31 00];
					max_hdr_ratio = [31 00];
					framerate_factor = "1000000";
					min_framerate = "2500000";
					max_framerate = "33000000";
					step_framerate = [31 00];
					default_framerate = "33000000";
					exposure_factor = "1000000";
					min_exp_time = "450";
					max_exp_time = "400000";
					step_exp_time = [31 00];
					default_exp_time = "16667";
					embedded_metadata_height = [31 00];
					set_mode_delay_ms = "500";
				};

				mode1 {
					mclk_khz = "37125";
					num_lanes = [32 00];
					tegra_sinterface = "serial_a";
					phy_mode = "DPHY";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = [30 00];
					dynamic_pixel_bit_depth = "12";
					csi_pixel_bit_depth = "12";
					mode_type = "bayer";
					pixel_phase = "rggb";
					active_w = "1920";
					active_h = "1080";
					readout_orientation = [30 00];
					line_length = "1036";
					inherent_gain = [31 00];
					mclk_multiplier = "25.6";
					pix_clk_hz = "950400000";
					gain_factor = "10";
					min_gain_val = [30 00];
					max_gain_val = "300";
					step_gain_val = [33 00];
					default_gain = [30 00];
					min_hdr_ratio = [31 00];
					max_hdr_ratio = [31 00];
					framerate_factor = "1000000";
					min_framerate = "2500000";
					max_framerate = "60000000";
					step_framerate = [31 00];
					default_framerate = "60000000";
					exposure_factor = "1000000";
					min_exp_time = "450";
					max_exp_time = "400000";
					step_exp_time = [31 00];
					default_exp_time = "16666";
					embedded_metadata_height = [31 00];
					set_mode_delay_ms = "500";
				};

				ports {
					#address-cells = <0x1>;
					#size-cells = <0x0>;

					port@0 {
						reg = <0x0>;

						endpoint {
							port-index = <0x0>;
							bus-width = <0x2>;
							remote-endpoint = <0xcb>;
							linux,phandle = <0x79>;
							phandle = <0x79>;
						};
					};
				};
			};

Unfortunately it did not help. Did i set “set_mode_delay_ms” correctly?

Thank you
Timo

I don’t think it’s similar issue.
Your issue could be relative about the MIPI timing cause capture failed.
You can try modify the discontinuous_clk in device tree to try.
BTW check with sensor vendor to confirm the sensor output clock is continuous or discontinuous to modify the discontinuous_clk.

Thank you for your support @ShaneCCC !

Unfortunately modifying the discontinous_clk did not help. After setting it to “yes” the camera would not work at all.

Today we were able to get some additional Log-Info out of the system, that we did not see before:

[  +1,518308] fence timeout on [ffffffc08855b480] after 1500ms
[  +0,000008] name=[nvhost_sync:17], current value=3538 waiting value=3539
[  +0,000005] ---- mlocks ----

[  +0,000011] ---- syncpts ----
[  +0,000007] id 1 (disp0_a) min 1204 max 1204 refs 1 (previous client : )
[  +0,000004] id 2 (disp0_b) min 2 max 2 refs 1 (previous client : )
[  +0,000003] id 3 (disp0_c) min 2 max 2 refs 1 (previous client : )
[  +0,000006] id 8 (gm20b_507) min 23670 max 23670 refs 1 (previous client : )
[  +0,000004] id 9 (gm20b_506) min 14 max 14 refs 1 (previous client : )
[  +0,000004] id 11 (gm20b_505) min 13534 max 13534 refs 1 (previous client : gm20b_505)
[  +0,000005] id 12 (54340000.vic_0) min 6333 max 6333 refs 1 (previous client : 54340000.vic_0)
[  +0,000006] id 17 (54680000.isp_0) min 3538 max 3541 refs 4 (previous client : 54680000.isp_0)
[  +0,000004] id 18 (54680000.isp_1) min 3538 max 3541 refs 4 (previous client : 54680000.isp_1)
[  +0,000003] id 19 (54680000.isp_2) min 11773 max 11781 refs 10 (previous client : 54680000.isp_2)
[  +0,000004] id 20 (54680000.isp_3) min 3539 max 3541 refs 4 (previous client : 54680000.isp_3)
[  +0,000003] id 21 (gm20b_504) min 704 max 704 refs 1 (previous client : )
[  +0,000004] id 22 (gm20b_503) min 58 max 58 refs 1 (previous client : )
[  +0,000003] id 23 (gm20b_502) min 2 max 2 refs 1 (previous client : )
[  +0,000003] id 24 (gm20b_501) min 2 max 2 refs 1 (previous client : )
[  +0,000003] id 25 (gm20b_500) min 6 max 6 refs 1 (previous client : )
[  +0,000004] id 26 (vblank0) min -2 max -2 refs 1 (previous client : )
[  +0,000004] id 28 (54340000.vic_0) min 1731 max 1731 refs 1 (previous client : 54340000.vic_0)
[  +0,000004] id 30 (54080000.vi_1) min 3538 max 3541 refs 4 (previous client : 54080000.vi_1)
[  +0,000004] id 31 (54080000.vi_2) min 7462 max 7467 refs 7 (previous client : 54080000.vi_2)
[  +0,000005] id 34 (54080000.vi_4) min 3539 max 3541 refs 4 (previous client : 54080000.vi_4)
[  +0,000003] id 35 (54340000.vic_0) min 7728 max 7728 refs 1 (previous client : 54340000.vic_0)
[  +0,000004] id 36 (54340000.vic_0) min 60 max 60 refs 1 (previous client : 54340000.vic_0)
[  +0,000004] id 37 (gm20b_497) min 2252 max 2252 refs 1 (previous client : gm20b_499)
[  +0,000003] id 38 (gm20b_495) min 2310 max 2310 refs 1 (previous client : gm20b_494)
[  +0,000004] id 39 (gm20b_496) min 2310 max 2310 refs 1 (previous client : gm20b_498)
[  +0,000004] id 40 (gm20b_499) min 2312 max 2312 refs 1 (previous client : gm20b_496)
[  +0,000003] id 41 (gm20b_498) min 2312 max 2312 refs 1 (previous client : gm20b_495)
[  +0,000003] id 42 (gm20b_494) min 2310 max 2310 refs 1 (previous client : gm20b_497)

[  +0,000089] ---- channels ----
[  +0,000008]
              channel 0 - 54680000.isp

[  +0,000006] 0-54680000.isp (18):
[  +0,000003] active class 01, offset 0050, val 00000011
[  +0,000003] DMAPUT 000001e0, DMAGET 00000158, DMACTL 00000000
[  +0,000002] CBREAD 00000011, CBSTAT 00010050
[  +0,000005]
              ffffffc07538f400: JOB, syncpt_id=17, syncpt_val=3539, first_get=000000d8, timeout=10000, num_slots=3, num_handles=1
[  +0,000009]     GATHER at e8f70000+4490, 8 words
[  +0,000002] 20000001
[  +0,000002] 00000411
[  +0,000002] 20000001
[  +0,000002] 00000512
[  +0,000002] 20000001
[  +0,000002] 00000614
[  +0,000002] 200c0001
[  +0,000002] 00000005

[  +0,000005]     GATHER at e8f70000+44b0, 2 words
[  +0,000002] 20000001
[  +0,000002] 00000013


[  +0,000006]
              channel 1 - 54080000.vi

[  +0,000005] 1-54080000.vi (12):
[  +0,000003] active class 01, offset 0050, val 0000001e
[  +0,000003] DMAPUT 00000ff8, DMAGET 00000ff8, DMACTL 00000000
[  +0,000002] CBREAD 0000001e, CBSTAT 00010050
[  +0,000005]
             ffffffc0821e4c00: JOB, syncpt_id=30, syncpt_val=3539, first_get=00000f10, timeout=10000, num_slots=7, num_handles=1
[  +0,000005]     GATHER at eeeb0000+0000, 35 words
[  +0,000002] 10620001
[  +0,000002] 0000001f
[  +0,000002] 10630002
[  +0,000002] 00000000
[  +0,000002] 00000000
[  +0,000002] 103a0001
[  +0,000002] 10100010
[  +0,000002] 103c0001
[  +0,000002] 10101010
[  +0,000002] 10420001
[  +0,000001] 00000001
[  +0,000002] 10430006
[  +0,000002] 00000000

I do think it is related to our problem.

Thank you!
Timo

Did you config the sensor output as continuous/discontinuous too?

Sorry for the dumb question, but where do i set the sensor output to continuous/discontinuous?

Please consult with sensor vendor for the REG configure.

Thanks

@ShaneCCC i reached out to the manufacturer, and they are checking the issue.

For some reason, i am able to do ~1800 Iterations on another system with an NVIDIA Jetson with EMMC-Storage (everything else is the same). Any Idea why there is such a big difference to the system with SD-Card?

Best Ragards
Timo

Does the EMMC Nano able boot from the SD-Card to check?

Unfortunately it does not have an SD-Card Slot. The software image is the same and has been flashed with ./flash.sh one time to sd-card and one time to emmc.

Also i tried the sd-card with several different Jetsons.

Thank You
Timo

Is there any updatefrom vendor side? Still an issue? Thanks

@kayccc @ShaneCCC Thank you for the support. We got in touch with the vendor, and they provided us with a new driver and dts revision where discontinuous_clk=yes is working, and the camera has been working since then without any issue.

So @ShaneCCC suggestion was the right one (you know your stuff!), and I was able to lead the vendor into the right direction. Since we are using different imx485 cameras at the moment, we were also able to patch the other drivers with the provided update.

So for anyone with an imx485 camera and the same issue here is what to do:

In the DTS you need to edit:

- discontinuous_clk = no;
+ discontinuous_clk = yes;

In the driver add the following line:

{0x3A53, 0x01},
+ {0x3D00, 0x11},

Thank you
Timo

2 Likes

It’s great to hear the camera working well on your project.
Thanks for the update and sharing with the community.