Hello,
I got titled error with 6 of IMX490/30fps cameras using custom libargus-dependent application .
This applicaition runs for several hours, but eventually stopped with syslog message as follows.
kernel: [11495.191616] tegra194-isp5 14800000.isp: wait for capture status failed
kernel: [11495.191633] tegra194-isp5 14800000.isp: isp capture get status failed
kernel: [11495.983292] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 2
kernel: [11495.987495] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 1
kernel: [11495.994183] imx490 31-001b: imx490_power_off: power off
kernel: [11495.999735] imx490 30-001c: imx490_power_off: power off
kernel: [11496.004953] imx490 30-001b: imx490_power_off: power off
kernel: [11496.011311] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 0
kernel: [11496.022313] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 5
kernel: [11496.025938] imx490 32-001c: imx490_power_off: power off
kernel: [11496.031293] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 3
kernel: [11496.039823] imx490 31-001c: imx490_power_off: power off
kernel: [11496.048285] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 4
kernel: [11496.056254] imx490 32-001b: imx490_power_off: power off
systemd[1]: nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
systemd[1]: nvargus-daemon.service: Failed with result ‘signal’.
4pcs of IMX490/30fps can last more than 12 hours.
I think this might be performance limit of isp or vi.
Is there any remedy to avoid it?
Runtime ver Jetpack 4.6.4.
Thank you.
hello hirofumi_momose,
here’re couple ways for issue narrow down.
(1) please try following commands to boost all the VI/CSI/ISP clocks.
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
(2) please test with Applications Using V4L2 IOCTL Directly instead of customize application.
Hello JerryChang,
(1) Yes, I already tried this boostup trick. But it does not solve the issue.
(2) Try, but get an error saying
VIDIOC_STREAMON: failed: Operation not permitted.
Best regards,
Hiro
hello hirofumi_momose,
v4l2 IOCTL it’s low-level test utility to check sensor functionality.
please refer to Jetson Xavier NX Product Design Guide for [Figure 9-2. CSI Connection Options] to double check your camera connections.
and… please confirm this issue is always related to number of camera streams.
Hello JerryChang,
v4l2-ctl works fine. Previous attempt was something wrong.
I can reproduce this issue using gstreamer shown below.
for i in seq 0 5
do
echo ---------------------------$i-------------------------
gst-launch-1.0 nvarguscamerasrc sensor-id=$i ! ‘video/x-raw(memory:NVMM),width=(int)2880,height=(int)1860,framerate=30/1’ ! fakesink sync=false &
sleep 10
done
Simultaneous 6 cams at same time will eventuarlly fail, but 4 cams works.
Best regards,
Hiro
hello hirofumi_momose,
did it meant you’re able to launch 6-camera stream, but there’s issue with the stream stability?
please share the complete failure logs for reference,
besides,
(1) what’s your environment settings, did you have thermal solution (such as fan cooling)?
(2) what’s the background running service? for instance, did you running with CPU/GPU resources fully occupied?
Hello JerryChang,
Yes, 6 cams works at first, but fails after a while.
(Time to failure varies from less than an hour to several.)
Here is syslog
Aug 1 20:07:44 nvidia-desktop kernel: [ 7505.916583] tegra194-isp5 14800000.isp: wait for capture status failed
Aug 1 20:07:44 nvidia-desktop kernel: [ 7505.916594] tegra194-isp5 14800000.isp: isp capture get status failed
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.683477] imx490 32-001c: imx490_power_off: power off
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.689792] imx490 32-001b: imx490_power_off: power off
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.696053] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 3
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.701938] imx490 31-001c: imx490_power_off: power off
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.709025] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 5
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.716060] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 2
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.718831] imx490 31-001b: imx490_power_off: power off
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.730040] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 1
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.735061] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 4
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.743357] imx490 30-001c: imx490_power_off: power off
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.749037] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 0
Aug 1 20:07:44 nvidia-desktop kernel: [ 7506.763317] imx490 30-001b: imx490_power_off: power off
Aug 1 20:07:45 nvidia-desktop systemd[1]: nvargus-daemon.service: Main process exited, code=killed, status=11/SEGV
Aug 1 20:07:45 nvidia-desktop systemd[1]: nvargus-daemon.service: Failed with result ‘signal’.
Aug 1 20:07:45 nvidia-desktop systemd[1]: nvargus-daemon.service: Service hold-off time over, scheduling restart.
Aug 1 20:07:45 nvidia-desktop systemd[1]: nvargus-daemon.service: Scheduled restart job, restart counter is at 1.
Aug 1 20:07:45 nvidia-desktop systemd[1]: Stopped Argus daemon.
Aug 1 20:07:45 nvidia-desktop systemd[1]: Started Argus daemon.
(1) following images are captured with 6-cams.
fan cooling works.
(2) No major process is running except for gstreamer and nvargus-daemon.
Best regars,
Hiro.
hello hirofumi_momose,
it looks you’re running with 6 gst pipeline, could you please also adding timeout options for confirmation.
for instance,
$ gst-launch-1.0 nvarguscamerasrc timeout=60 sensor-id=$i ! ‘video/x-raw(memory:NVMM),width=(int)2880,height=(int)1860,framerate=30/1’ ! fakesink sync=false &
BTW,
we would like to confirm what’s happening to cause wait for capture status failed.
it’s suggest you setting-up a terminal with $ dmesg --follow to keep gather the logs simultaneously.
Hello JerryChang,
Adding timeout=60 finished without error.
This problem will take long time to occur, difficult to reproduce within 60 seconds.
And is there any difference between dmsg after errror and dmesg --follow ?
dmesg result is very close to syslog.
hello hirofumi_momose,
it seems some logs has truncated,
please share complete logs for digging into root cause.
thanks
Hello JerryChang,
dmesg --follow is live log printing, not reveal more details.
instead, I did followings.
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
so many log below is appeard.
[ 1953.337157] rtcpu_trace_isp_event event id 20 cannot be found
removing that log and imx490 gain/exposure? controls like:
[ 209.893047] imx490 30-001b: imx490_set_coarse_time: coarse1:1441, shs1:1441, FL:3600
[ 209.918563] imx490 30-001b: imx490_set_gain: gain:1175
[ 209.922604] imx490 30-001b: imx490_set_coarse_time: coarse1:1996, shs1:1996, FL:3600
[ 209.951813] imx490 30-001b: imx490_set_gain: gain:1251
finally, obtained
[ 206.088311] imx490 30-001b: imx490_open:
[ 206.089027] imx490 32-001c: imx490_open:
[ 206.089688] imx490 30-001c: imx490_open:
[ 206.090377] imx490 31-001b: imx490_open:
[ 206.091000] imx490 31-001c: imx490_open:
[ 206.091705] imx490 32-001b: imx490_open:
[ 206.860485] [RCE] Configuring ISP GoS.
[ 206.860503] [RCE] VM GOS[#0] addr=0xc2100000
[ 206.860517] [RCE] VM GOS[#1] addr=0xc2101000
[ 206.860562] [RCE] VM GOS[#2] addr=0xc2102000
[ 206.860575] [RCE] VM GOS[#3] addr=0xc2103000
[ 206.860587] [RCE] VM GOS[#4] addr=0xc2104000
[ 206.860599] [RCE] VM GOS[#5] addr=0xc2105000
[ 207.536466] [RCE] Configuring VI GoS.
[ 207.536488] [RCE] VM GOS[#0] addr=0xc2100000
[ 207.536500] [RCE] VM GOS[#1] addr=0xc2101000
[ 207.536512] [RCE] VM GOS[#2] addr=0xc2102000
[ 207.536524] [RCE] VM GOS[#3] addr=0xc2103000
[ 207.536535] [RCE] VM GOS[#4] addr=0xc2104000
[ 207.536547] [RCE] VM GOS[#5] addr=0xc2105000
[ 207.536564] [RCE] vi5_hwinit: firmware CL2018101701 protocol version 2.2
[ 207.872580] [RCE] Configuring VI GoS.
[ 207.872606] [RCE] VM GOS[#0] addr=0xc2100000
[ 207.872621] [RCE] VM GOS[#1] addr=0xc2101000
[ 207.872634] [RCE] VM GOS[#2] addr=0xc2102000
[ 207.872647] [RCE] VM GOS[#3] addr=0xc2103000
[ 207.872659] [RCE] VM GOS[#4] addr=0xc2104000
[ 207.872674] [RCE] VM GOS[#5] addr=0xc2105000
[ 207.879146] imx490 30-001b: imx490_power_on: power on
[ 209.108451] [RCE] VI GOS[#0] set to VM GOS[4] base 0xc2104000
[ 209.276514] [RCE] ISP GOS[#0] set to VM GOS[5] base 0xc2105000
[ 216.236519] [RCE] Configuring VI GoS.
[ 216.236551] [RCE] VM GOS[#0] addr=0xc2100000
[ 216.236562] [RCE] VM GOS[#1] addr=0xc2101000
[ 216.236573] [RCE] VM GOS[#2] addr=0xc2102000
[ 216.236583] [RCE] VM GOS[#3] addr=0xc2103000
[ 216.236602] [RCE] VM GOS[#4] addr=0xc2104000
[ 216.236615] [RCE] VM GOS[#5] addr=0xc2105000
[ 216.463518] imx490 30-001c: imx490_power_on: power on
[ 216.464599] [RCE] Configuring VI GoS.
[ 216.464616] [RCE] VM GOS[#0] addr=0xc2100000
[ 216.464638] [RCE] VM GOS[#1] addr=0xc2101000
[ 216.464651] [RCE] VM GOS[#2] addr=0xc2102000
[ 216.464658] [RCE] VM GOS[#3] addr=0xc2103000
[ 216.464674] [RCE] VM GOS[#4] addr=0xc2104000
[ 216.464684] [RCE] VM GOS[#5] addr=0xc2105000
[ 1953.319113] tegra194-isp5 14800000.isp: wait for capture status failed
[ 1953.319122] tegra194-isp5 14800000.isp: isp capture get status failed
[ 1954.036233] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 5
[ 1954.038915] imx490 32-001c: imx490_power_off: power off
[ 1954.046233] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 4
[ 1954.054215] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 2
[ 1954.057971] imx490 32-001b: imx490_power_off: power off
[ 1954.063203] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 3
[ 1954.070312] imx490 31-001c: imx490_power_off: power off
[ 1954.074868] imx490 31-001b: imx490_power_off: power off
[ 1954.078230] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 1
[ 1954.083678] imx490 30-001c: imx490_power_off: power off
[ 1954.087219] tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 0
[ 1954.098094] imx490 30-001b: imx490_power_off: power off
Umm, nothing improved.
It is possible that log might be snipped because of so many “rtcpu_trace_isp_event event id 20 cannot be found”.
Regards,
hello hirofumi_momose,
you might gather all the logs as single text file, and attach the file for reference.
Hello JerryChang,
2 archives attached,
each contains
gstreamer console output
syslog
dmesg
wo_debuglog.tar.gz (217.8 KB)
w_debuglog.tar.gz (10.6 MB)
Thank you.
hello hirofumi_momose,
please try replace libgstnvarguscamerasrc.so with the attachment, Topic300209_Aug06.7z (25.7 KB)
it’s based-on JP-4.6.4/r32.7.4 release version for adding configurable timeout property, such as event-wait and acquire-wait to the gst pipeline.
you may see-also $ gst-inspect-1.0 nvarguscamerasrc for property descriptions.
here’s sample command-line for your reference,
$ gst-launch-1.0 -v nvarguscamerasrc event-wait=5000000000 acquire-wait=6000000000 sensor-id=0 ! video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080,framerate=30/1 ! nvvidconv ! fakesink -e
BTW,
(1) there’re some Invalid isp config attribute had reported,
please removing those ISP settings to avoid such warning, and potential failures.
(2) according to the logs…
it shows pixel incomplete error, which may due to your sensor stream has failures intermittently. please double check your sensor configurations.
Aug 5 18:40:55 nvidia-desktop nvargus-daemon[4789]: NvCaptureStatusErrorDecode Stream 2.0 failed: sof_ts 127681862944 eof_ts 4086603867136 frame 0 error 14 data 0x00020000
Aug 5 18:40:55 nvidia-desktop nvargus-daemon[4789]: NvCaptureStatusErrorDecode Capture-Error: FALCON_ERROR (0x0000000e)
Aug 5 18:40:55 nvidia-desktop nvargus-daemon[4789]: SCF: Error InvalidState: Capture error with status 14 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 953)
Hello JerryChang,
Unfortuntely, replacing .so file does not work.
I think this is caused by prior stage, likely isp time out triggers nvargus-daemon to crash, finally gstreamer fails.
Can I access source code of “tegra194-isp5 14800000.isp: wait for capture status failed” ?
And (1)isp config error is fixed. (2) I am not sure of this error,
I ask driver manufacturer. But this error occurs at startup only. seems not root of the problem.
Thank you.
hello hirofumi_momose,
per your comments…
please also share your camera connections for reference,
for instance, what CSI ports you’re used for 6-cam setups.
Hello JerryChang,
Using LI-IMX490-GMSL2 from Leopard Imaging.
Machine is LI-XNX-BOX-GMSL2 also from Leopard.
All cameras are connected with GMSL2.
Additionally, another driver (20fps/No hdr/JP 4.5.1) works well with 6 cams.
Regards,
Hiro.
This issue will require our camera partner to do the support, let me add @xingxing here to help you.
@hirofumi_momose
We have the Xavier NX driver with 2880x1860@30fps. Please contact support@leopardimaging.com to get the driver guide.
