Nvargus-daemon does not start camera feed on Nano

vsaw · February 4, 2021, 3:34pm

Hi there,

since a few days ago, I can not get a camera feed anymore on my Jetson Nano.

Setup

Jetson Nano 4GB
JetPack 4.3 (R32 (release), REVISION: 3.1, GCID: 18186506, BOARD: t210ref, EABI: aarch64, DATE: Tue Dec 10 06:58:34 UTC 2019)
IMX219 Camera connected via CSI

Camera Pipeline

I use gst-launch1.0 to access the camera feed as follows

gst-launch-1.0 \
    nvarguscamerasrc sensor-id=0 ! \
    "video/x-raw(memory:NVMM), width=1280, height=720, format=(string)NV12, framerate=(fraction)120/1" ! \
    videorate max-rate=10 ! \
    nvvidconv flip-method=2 ! \
    jpegenc ! \
    rtpjpegpay ! \
    queue leaky=downstream max-size-buffers=1 ! \
    udpsink host=127.0.0.1 port=7001

The output of this command is

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21,000000 fps Duration = 47619048 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28,000001 fps Duration = 35714284 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59,999999 fps Duration = 16666667 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120,000005 fps Duration = 8333333 ; Analog Gain range min 1,000000, max 10,625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 120,000005
GST_ARGUS: PowerService: requested_clock_Hz=24192000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.

However my application receives no frames. This behavior is consistent across reboots.

Additional Information

Debug Logs

Looking at /var/log/sysctl I get a lot of error messages from nvargus-daemon. I’ve included a few of them in the attachment. Also dmesg complains a lot about syncpts.

dmsg.txt (18.9 KB)
syslog-nvargus-daemon.txt (8.1 KB)

Final Remarks

As mentioned in the beginning: The gstreamer pipeline was working until a few days ago. I am not aware of any changes in my setup that caused the camera to stop working.

An pointers on what I should try to fix this are greatly appreciated!

ShaneCCC · February 5, 2021, 3:30am

Have reference to below doc to verify sensor working well. If not that could be the HW or connection problem.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/camera_sensor_prog.html#wwpID0E0LF0HA

vsaw · February 5, 2021, 9:44am

Thanks @ShaneCCC for the quick reply. From what I can tell the HW connection is fine. v4l2-compliance does produce one failure, but I get the exact same output on another device where the camera is working fine.

$ v4l2-compliance -d /dev/video0
v4l2-compliance SHA   : not available

Driver Info:
	Driver name   : tegra-video
	Card type     : vi-output, imx219 7-0010
	Bus info      : platform:54080000.vi:0
	Driver version: 4.9.140
	Capabilities  : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format

Compliance test for device /dev/video0 (not using libv4l2):

Required ioctls:
	test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
	test second video open: OK
	test VIDIOC_QUERYCAP: OK
	test VIDIOC_G/S_PRIORITY: OK
	test for unlimited opens: OK

Debug ioctls:
	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
	test VIDIOC_LOG_STATUS: OK

Input ioctls:
	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
	test VIDIOC_ENUMAUDIO: OK (Not Supported)
	test VIDIOC_G/S/ENUMINPUT: OK
	test VIDIOC_G/S_AUDIO: OK (Not Supported)
	Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
	Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
	test VIDIOC_G/S_EDID: OK (Not Supported)

Test input 0:

	Control ioctls:
		test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
		test VIDIOC_QUERYCTRL: OK
		test VIDIOC_G/S_CTRL: OK
		test VIDIOC_G/S/TRY_EXT_CTRLS: OK
		test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
		test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
		Standard Controls: 1 Private Controls: 16

	Format ioctls:
		test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
		fail: v4l2-test-formats.cpp(1184): ret && node->has_frmintervals
		test VIDIOC_G/S_PARM: FAIL
		test VIDIOC_G_FBUF: OK (Not Supported)
		test VIDIOC_G_FMT: OK
		test VIDIOC_TRY_FMT: OK
		test VIDIOC_S_FMT: OK
		test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
		test Cropping: OK (Not Supported)
		test Composing: OK (Not Supported)
		test Scaling: OK (Not Supported)

	Codec ioctls:
		test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
		test VIDIOC_G_ENC_INDEX: OK (Not Supported)
		test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

	Buffer ioctls:
		test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
		test VIDIOC_EXPBUF: OK

Test input 0:


Total: 43, Succeeded: 42, Failed: 1, Warnings: 0

Whereas v4l2-ctl seems to get stuck on the following output:

$ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG12 --stream-mmap --set-ctrl=sensor_mode=0 --stream-count=100 -d /dev/video0 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
	Width/Height      : 3264/2464
	Pixel Format      : 'RG12'
	Field             : None
	Bytes per Line    : 6528
	Size Image        : 16084992
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Full Range)
	Flags             :
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok

Update

I thought that CSI cameras are not available on v4l2. See for example: Jetpack 4.5 CSI cam can't use with v4l2 /dev/video0 - #3 by JerryChang

ShaneCCC · February 5, 2021, 9:58am

Check with this command. And enable the debug print of the csi2_fops.c to check the status.

v4l2-ctl --stream-mmap --stream-count=100 --set-ctrl bypass_mode=0 -d /dev/video0

vsaw · February 5, 2021, 10:33am

Thank you @ShaneCCC .

I’m not sure how to enable the debug prints, but running the command directly produces the following output:

First Try

$ v4l2-ctl --stream-mmap --stream-count=100 --set-ctrl bypass_mode=0 -d /dev/video0 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_REQBUFS: failed: Device or resource busy

Second Try

I then rebooted the device and then it freezes with the following output:

$ v4l2-ctl --stream-mmap --stream-count=100 --set-ctrl bypass_mode=0 -d /dev/video0 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok

On a different device with the same HW I get

$ v4l2-ctl --stream-mmap --stream-count=100 --set-ctrl bypass_mode=0 -d /dev/video0
<<<<<<<<<<<<<<<<<<<<<<< 21.15 fps
<<<<<<<<<<<<<<<<<<<<< 21.07 fps
<<<<<<<<<<<<<<<<<<<<< 21.05 fps
<<<<<<<<<<<<<<<<<<<<< 21.03 fps
<<<<<<<<<<<<<<

If you give me any pointers on how to enable the debug prints I’d be happy to try again.

ShaneCCC · February 5, 2021, 12:15pm

Run below command then run the v4l2-ctl and check the dmesg.

sudo su
echo file csi2_fops.c +p > /sys/kernel/debug/dynamic_debug/control

vsaw · February 5, 2021, 3:02pm

@ShaneCCC Done. Here’s what I have

$ v4l2-ctl --stream-mmap --stream-count=100 --set-ctrl bypass_mode=0 -d /dev/video0 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_REQBUFS: failed: Device or resource busy

$ dmesg | tail -n 3
[ 2252.125134] tegra-vii2c 546c0000.i2c: no acknowledge from address 0x10
[ 2252.131912] regmap_util_write_table_8:regmap_util_write_table:-121
[ 2252.188628] imx219 7-0010: Error turning off streaming

Larger dmesg output attached: dmesg-device-busy.txt (5.0 KB)

After restarting `nvargus-daemon`

I restarted nvargus-daemon and ran v4l2-ctl again with the following dmesg output

[16301.316395] vi 54080000.vi: Calibrate csi port 0
[16301.415171] vi 54080000.vi: cil_settingtime was autocalculated
[16301.415178] vi 54080000.vi: csi clock settle time: 13, cil settle time: 10
[16301.645395] video4linux video0: frame start syncpt timeout!0
[16301.651205] vi 54080000.vi: TEGRA_CSI_PIXEL_PARSER_STATUS 0x00000000
[16301.651225] vi 54080000.vi: TEGRA_CSI_CIL_STATUS 0x00000010
[16301.651242] vi 54080000.vi: TEGRA_CSI_CILX_STATUS 0x00040041
[16301.651343] vi 54080000.vi: cil_settingtime was autocalculated
[16301.651363] vi 54080000.vi: csi clock settle time: 13, cil settle time: 10
[16301.853742] video4linux video0: frame start syncpt timeout!0
[16301.859882] vi 54080000.vi: TEGRA_CSI_PIXEL_PARSER_STATUS 0x00000000
[16301.859933] vi 54080000.vi: TEGRA_CSI_CIL_STATUS 0x00000010
[16301.859980] vi 54080000.vi: TEGRA_CSI_CILX_STATUS 0x00040041
[16301.860192] vi 54080000.vi: cil_settingtime was autocalculated
[16301.860245] vi 54080000.vi: csi clock settle time: 13, cil settle time: 10

The messages just keep repeating the last lines a few more times.

Working Device

For a device with the same HW produces the following output.

[101761.187185] vi 54080000.vi: Calibrate csi port 0
[101761.290657] vi 54080000.vi: cil_settingtime was autocalculated
[101761.290664] vi 54080000.vi: csi clock settle time: 13, cil settle time: 10
[101766.384606] video4linux video0: tegra_channel_capture_done: MW_ACK_DONE syncpoint time out!0
[101766.393165] vi 54080000.vi: TEGRA_CSI_PIXEL_PARSER_STATUS 0x00000080
[101766.393170] vi 54080000.vi: TEGRA_CSI_CIL_STATUS 0x00000000
[101766.393175] vi 54080000.vi: TEGRA_CSI_CILX_STATUS 0x00000000
[101766.393219] vi 54080000.vi: cil_settingtime was autocalculated
[101766.393224] vi 54080000.vi: csi clock settle time: 13, cil settle time: 10

vsaw · February 11, 2021, 4:05pm

@ShaneCCC do you still need more infos from me? I’m still lost so any help is greatly appreciated.

ShaneCCC · February 12, 2021, 2:13am

I think need HW engineer help to check what’s the different for the failed device.

vsaw · February 15, 2021, 5:10pm

It looks like it was a HW error after all. With a different CSI-Cable the device starts up again. (I was only able to check this after the camera has been returned from the customer)

But I don’t know I was able to enumerate the Camera Resolution and other information with the broken cable in the first place 🤷‍♂️

Thank you @ShaneCCC for your support.