Argus API is not stable in JetPack 5 and gstreamer doesnt work

Hello all,

I ported a camera driver GMSL2 - MIPI CSI 2 from JetPack 4 to JetPack 5.0.2 (L4T 35.1)
The capture works perfectly using v4l2 ctl, for example with using the command below, it queue and dequeu buffer successfully and it prints the fps

v4l2-ctl --stream-mmap

But when tesing with argus samples like oneShot and other, once it works and once it get stuck and prints

Failed to getIFrameInterface

And then i need to kill the nvargus-daemon process, but it keeps not working

The only solution i found is the following

# Kill nvargus-daemon
$ sudo kill <nvargus-daemon process>

# Launch nvargus-daemon manually when the API get stuck
$ sudo nvargus-daemon 

And the when i launch the argus samples it works

By the way the gstreamer nvarguscamerasrc doesn’t work also in jetpack 5 and fails with error bellow

jetson@jetson-desktop:~/jetson_multimedia_api/samples/13_multi_camera$ gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)3848, height=(int)2168, format=(string)NV12, framerate=(fraction)30/1' ! nvvidconv ! queue ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3848 x 2168 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.000000; Exposure Range min 15000, max 33333000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 15.000000; Exposure Range min 15000, max 33333000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3848 H = 2168 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:694 NvBufSurfaceFromFd Failed.
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:247 (propagating)
Got EOS from element "pipeline0".
Execution ended after 0:00:04.006677128
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up



Can you please suggest some solution to make the argus api works properly ?

Thanks

For gstreamer, i solved the issue using the command below

export DISPLAY=:0

But the kill of the argus process like the following is still required

jetson@jetson-desktop:~$ ps aux | grep argus
root        1397  0.6  0.9 17283020 138992 ?     Ssl  09:44   0:01 /usr/sbin/nvargus-daemon
jetson      3311  0.0  0.0   8572   664 pts/1    S+   09:47   0:00 grep --color=auto argus
jetson@jetson-desktop:~$ sudo kill nvargus-daemon
kill: failed to parse argument: 'nvargus-daemon'
jetson@jetson-desktop:~$ sudo kill 1397
jetson@jetson-desktop:~$ sudo nvargus-daemon

You can see that when launching nvargus-daemon manually it’s working

Is this issue related to JetPack 5 ?

Thanks

Does “sudo service nvargus-daemon restart” working?

Hi @ShaneCCC

Yes, it works but, when i ran an application for several times, it fails.
I’m getting

Failed to get IFrame interface

And when i try to run again the app, the argus init fails

Failed to create capture session

Of course, i restart nvargus-daemon

sudo service nvargus-daemon restart

The app works once or two times and then it get stuck…

There is some logs :
In a terminal 1 a ran the oneshot app

jetson@jetson-desktop:~/jetson_multimedia_api/argus/build/samples/oneShot$ ./argus_oneshot 
Executing Argus Sample: argus_oneshot
Argus Version: 0.98.3 (multi-process)
Capturing from device 0 using sensor mode 0 (3848x2168)
Failed to get IFrame interface
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
(Argus) Error InvalidState: Argus client is exiting with 2 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 355)

In a terminal 2

(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
(NvCamV4l2) Error ModuleNotPresent:  (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 3. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 646)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 437)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 311)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 477)
Acquiring SCF Camera device source via index 1 has failed. === argus_oneshot[3964]: CameraProvider initialized (0xffff746173f0)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
SCF: Error InvalidState: 1 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 302)
SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 0 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 507)
SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error InvalidState: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 593)
SCF: Error Timeout:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
SCF: Error Timeout: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
waitForIdleLocked remaining request 101 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 909)
(Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
waitForIdleLocked remaining request 101 
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 909)
SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 875)
SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 405)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)

There was not such an issue in Jetpack 4.x, is this related to the new version if Jetpack ?
I’m using the 5.0.2 GA, is this bug resolved in the lattest version 5.1.1 ?

Thanks

Does your sensor need much time for i2c REG initial?
If yes try to reduce it.

@ShaneCCC do you think is related to the i2c initi ? Because v4l2 ctl works perfectly.
In JetPack 4.x i am using the same time for i2c reg for the image sensor.

By the way i am using a setup with max9295, max9296 serdes pair with ar0820 as image sensor.

Have i to reduce the time of i2c reg in ar0820 driver like in the write function ?

static int ar0820_write_reg(struct camera_common_data *s_data,
			    u16 addr, u16 val)
{
	int err;
	struct device *dev = s_data->dev;
	err = regmap_write(s_data->regmap, addr, val);

	if (err) {
		dev_err(dev, "%s:i2c write failed, 0x%x = %x\n",
			__func__, addr, val);
	}
	usleep_range(1000, 1010);
	return err;
}

Thanks

@ShaneCCC i reduced the i2c init time, but still the issue, with Argus, the capture works 1 / 5, when it get stuck i have to restart the nvargus-daemon so Argus can create a capture sesssion…
I think the issue is really related to the Argus API handeling…

There is some logs related to Argus

jetson@jetson-desktop:~$ sudo journalctl -u nvargus-daemon --follow 
-- Logs begin at Thu 2022-04-21 08:54:39 EDT. --
May 08 13:30:41 jetson-desktop nvargus-daemon[3611]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 909)
May 08 13:30:41 jetson-desktop nvargus-daemon[3611]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 875)
May 08 13:30:41 jetson-desktop nvargus-daemon[3611]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 405)
May 08 13:30:43 jetson-desktop nvargus-daemon[3611]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
May 08 13:30:43 jetson-desktop nvargus-daemon[3611]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
May 08 13:30:43 jetson-desktop nvargus-daemon[3611]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
May 08 13:34:09 jetson-desktop systemd[1]: Stopping Argus daemon...
May 08 13:34:09 jetson-desktop systemd[1]: nvargus-daemon.service: Succeeded.
May 08 13:34:09 jetson-desktop systemd[1]: Stopped Argus daemon.
May 08 13:34:09 jetson-desktop systemd[1]: Started Argus daemon.
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: === NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== oneShot[4285]: Connection established (FFFFAC43A900)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: NvPclHwGetModuleList: No module data found
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: NvPclHwGetModuleList: No module data found
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: E/ libnvphs:socket: Error[2]: socket connection /var/lib/nvphs/nvphsd.ctl to PHS failed: No such file or directory
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: D/ libnvphs:socket: Warning: connecting to Power Hinting Service failed. Is PHS running?
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: === oneShot[4285]: CameraProvider initialized (0xffffa45a0cb0)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: E/ libnvphs:socket: Error[2]: socket connection /var/lib/nvphs/nvphsd.ctl to PHS failed: No such file or directory
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: D/ libnvphs:socket: Warning: connecting to Power Hinting Service failed. Is PHS running?
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=MinCPU, value=4294967295, timeout_ms=1000]: queue_or_send() failed
May 08 13:34:48 jetson-desktop nvargus-daemon[3942]: CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
May 08 13:34:53 jetson-desktop nvargus-daemon[3942]: SCF: Error InvalidState: 1 buffers still pending during EGLStreamProducer destruction (propagating from src/services/gl/EGLStreamProducer.cpp, function freeBuffers(), line 302)
May 08 13:34:53 jetson-desktop nvargus-daemon[3942]: SCF: Error InvalidState:  (propagating from src/services/gl/EGLStreamProducer.cpp, function ~EGLStreamProducer(), line 50)
May 08 13:34:54 jetson-desktop nvargus-daemon[3942]: SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 0 (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 507)
May 08 13:34:54 jetson-desktop nvargus-daemon[3942]: SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
May 08 13:34:54 jetson-desktop nvargus-daemon[3942]: SCF: Error InvalidState: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
May 08 13:34:54 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 593)
May 08 13:34:54 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/common/Utils.cpp, function workerThread(), line 114)
May 08 13:34:54 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout: Worker thread ViCsiHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
May 08 13:34:54 jetson-desktop nvargus-daemon[3942]: Module_id 30 Severity 2 : (fusa) Error: Timeout  propagating from:/capture/src/fusaViHandler.cpp 776
May 08 13:34:57 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
May 08 13:34:57 jetson-desktop nvargus-daemon[3942]: Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
May 08 13:34:58 jetson-desktop nvargus-daemon[3942]: waitForIdleLocked remaining request 101
May 08 13:34:58 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 909)
May 08 13:34:58 jetson-desktop nvargus-daemon[3942]: (Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)
May 08 13:35:00 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
May 08 13:35:00 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
May 08 13:35:00 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)
May 08 13:35:04 jetson-desktop nvargus-daemon[3942]: waitForIdleLocked remaining request 101
May 08 13:35:04 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 909)
May 08 13:35:04 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function abortCaptures(), line 875)
May 08 13:35:04 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/api/Session.cpp, function shutdown(), line 405)
May 08 13:35:05 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 368)
May 08 13:35:05 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 936)
May 08 13:35:05 jetson-desktop nvargus-daemon[3942]: SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1029)

Could you check if the output data rate > 1.5Gbps?
If yes could you try have more delay after sensor stream on REG wrote.

Skew calibration is required if sensor or deserializer is using DPHY, and the output data rate is > 1.5Gbps.
An initiation deskew signal should be sent by sensor or deserializer to perform the skew calibration. If the deskew signals is not sent, the receiver will stall, and the capture will time out.
You can calculate the output data rate with the following equation:

Output data rate = (sensor or deserializer pixel clock in hertz) * (bits per pixel) / (number of CSI lanes)

@ShaneCCC

Have a look on the mode properties i am using (This works perfectly for Jetpack 4.x), it seems it’s > 1.5 Gbs

				mode0 {/*mode AR0820_MODE_3848X2168_30FPS*/
					mclk_khz = "24000";
					num_lanes = "4";
					tegra_sinterface = "serial_a";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = "0";

					dynamic_pixel_bit_depth = "12";
					csi_pixel_bit_depth = "12";
					mode_type = "bayer";
					pixel_phase = "grbg";

					active_w = "3848";
					active_h = "2168";
					readout_orientation = "0";
					line_length = "4440";
					inherent_gain = "1";
					mclk_multiplier = "14.58";
					pix_clk_hz = "296236800";
					serdes_pix_clk_hz = "833333333";

					gain_factor = "1";
					min_gain_val = "1";
					max_gain_val = "15";
					step_gain_val = "1";
					default_gain = "1";
					min_hdr_ratio = "1";
					max_hdr_ratio = "1";
					framerate_factor = "1000000";
					min_framerate = "30000000";
					max_framerate = "30000000";
					step_framerate = "1";
					default_framerate ="30000000";
					exposure_factor ="1000000";
					min_exp_time = "15";
					max_exp_time = "33333";
					step_exp_time = "1";
					default_exp_time = "33333";

					embedded_metadata_height = "0";
				};

What i’m not understanding is why the Argus API producer has to be restart when the capture fails, is it a bug in JetPack 5.x ?

Thanks

Yes, current J5.x add the deskew calibration feature cause the > 1.5Gbps have this kind of problem.
Try to modify serdes_pix_clk_hz to try. May need boost the clock manually for debug.

1 Like

Making a suitable value of serdes_pix_clk_hz and pix_clk_hz solved the issue.

pix_clk_hz = "74250000"
serdes_pix_clk_hz = "200000000"

Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.