Camera timing sensitiveness in JetPack 4.6.x

[diego.chaverri]

Hello,

We have been working for a while with a customer involving On-Semi’s AR0820 / AR0821 camera sensor, up to 12 cameras (equipped with MAX96705 serializer) going through 4 x MAX96712 deserializers (3 streams per deserializer) and finally to the Jetson CSI2 MIPI ports; the initial support for the driver was made based on JetPack 4.4 (L4T32.4.3), both for linear and HDR (3 exposures) capture modes with good results in terms of functionality / stability.

Recently, a porting of the driver was done to JetPack 4.6 (L4T32.6.1) which has shown to be quite time-sensitive, likely due to camera sensors with a slow-to-start sequence; from the very first testing cycle in L4T32.6.1, the following error was spotted (did not occur in L4T32.4.3) with a higher frequency in the case of the HDR capture using nvarguscamerasrc element:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 1 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 29,999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:00:02.276132993
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
Setting pipeline to NULL ...
Freeing pipeline ...

Now, our main concern is that there are a number of references to different sources to adjust the timeout threshold for the capture, however, it is not clear which one of the approaches listed below can be used to properly address the error shown:

1. Nvargus Daemon Infinite Timeout: The first and most common suggestion is to set the Infinite Timeout nvargus-daemon environment variable:

This first approach has shown to make no difference in the occurrence of the error.

2. First frame delay device tre property: This has been for a while in the developer guide, still it is not clear if this property is valid or not for L4T32.6.1 and under which circumstances can be used.

3. Modification to the timeout variables in the nvarguscamerasrc element: This approach basically points to modifying the WAIT_FOR_EVENT_TIMEOUT and ACQUIRE_FRAME_TIMEOUT nvarguscamerasrc element timeout values, please refer to the following link for further information:

https://forums.developer.nvidia.com/t/errors-starting-6-nvarguscamerasrc-pipelines-in-r32-5-0/170477

4. VI driver timeout: Finally, among the several alternative found in the forums, there is also the modification to the VI driver, specifically the CAPTURE_TIMEOUT_MS value defined in the vi5_fops.c file

So, given the information just shown,

• What is the use case for each of the timeout modification approaches shown above?
• Is there any additional information that we can provide to NVIDIA to debug the issue?
• Which is the suggested approach to debug the nvarguscamerasrc0: CANCELLED error? Is this actually a timeout error? Is it any different from the nvarguscamerasrc3: TIMEOUT error?

We appreciate your assistance on this matter as we have been dealing with it for a while, it is a blocking issue from the system stability point of view and it seems to have appeared after JetPack4.4 (we have references from developers having the same issue with JetPack 4.5.x and JetPack 4.6).

Regards
Diego Chaverri

[JerryChang]

hello diego.chaverri,

here’re known bug fixes for JetPack-4.6, please refer to below to include the fixes,
such as…

• Topic 187824, V4L2 timeout leads to NULL pointer dereference in kernel,
• Topic 184902, patch to fix “Invalid sensor placement is set” failure.

[DavidSoto-RidgeRun]

Hi Jerry,

Thanks for the quick response, we have tried both fixes but since we are using nvarguscamerasrc they seem not to help. They seem argus related. What would you suggest next?

[JerryChang]

hello DavidSoto-RidgeRun,

please enable VI trace for gathering more information for reference,
for example,

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace