VI performance issue with 6 AR0233 cameras on roscube-x?

after a minute or two it fails:

[ 287.315509] [RCE] WARNING: t194/isp5.c:901 [config_channel] “All error notifications not enabled: correctable=0x00 uncorrectable=0x00”
[ 287.987481] [RCE] vi5_hwinit: firmware CL2018101701 protocol version 2.2
[ 289.881163] max9295 30-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 435.370391] max9295 31-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 451.079362] max9295 31-0060: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 462.083808] max9295 32-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 473.147179] misc tegra_camera_ctrl: tegra_camera_update_isobw: Warning, Requested ISO BW 8138020 has been capped to VI’s max BW 8000000
[ 474.365791] max9295 32-0060: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 484.317677] misc tegra_camera_ctrl: tegra_camera_update_isobw: Warning, Requested ISO BW 9765624 has been capped to VI’s max BW 8000000
[ 485.539638] max9295 33-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 567.895451] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 567.895600] tegra194-vi5 15c10000.vi: vi capture get status failed
[ 567.923464] fence timeout on [ffffffc72a566f00] after 1500ms
[ 567.923475] name=[nvhost_sync:86], current value=6088 waiting value=6091
[ 567.923480] ---- mlocks ----

[ 567.923514] ---- syncpts ----
[ 567.923535] id 16 (gv11b_511) min 234 max 234 refs 1 (previous client : )
[ 567.923540] id 17 (gv11b_510) min 9 max 9 refs 1 (previous client : )
[ 567.923544] id 18 (gv11b_509) min 17 max 17 refs 1 (previous client : gv11b_509)
[ 567.923550] id 19 (15340000.vic_gst-launch-1.0_0) min 14517 max 14517 refs 1 (previous client : )
[ 567.923555] id 20 (15340000.vic_gst-launch-1.0_0) min 14493 max 14493 refs 1 (previous client : )
[…]

does the tegra_camera_update_isobw: Warning, Requested ISO BW 8138020 has been capped to VI's max BW 8000000 mean we are exceeding the VI’s band width with just 6 cameras?

all of the cameras are running in a plain 12bpp mode, except one of them, which is doing HDR+LFM. i would have expected to be able to run 8 cameras on this device concurrently, though…

also, quite annoying that situations like this seem to hose the VI/ISP/…? so that a hard reset (not just a reboot) is needed to recover.

The warnings may not be relevant here, I have also seen these in some working cases.

Not sure for your case, but you may try this.

the assumption that this happens due to the fancy HDR mode on the one camera seems wrong. it also stalls with 6 cameras in plain mode:

[ 118.691828] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[ 157.971178] [RCE] WARNING: t194/isp5.c:901 [config_channel] “All error notifications not enabled: correctable=0x00 uncorrectable=0x00”
[ 158.531181] [RCE] vi5_hwinit: firmware CL2018101701 protocol version 2.2
[ 159.954942] max9295 30-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 182.770821] max9295 31-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 193.895904] max9295 31-0060: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 206.003076] max9295 32-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 216.327022] misc tegra_camera_ctrl: tegra_camera_update_isobw: Warning, Requested ISO BW 8138020 has been capped to VI’s max BW 8000000
[ 217.553989] max9295 32-0060: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1
[ 230.506926] misc tegra_camera_ctrl: tegra_camera_update_isobw: Warning, Requested ISO BW 9765624 has been capped to VI’s max BW 8000000
[ 231.733386] max9295 33-0042: max9295_setup_streaming: Lane mapping: lane_map1 = 0XBE, lane_map2 = 0XE1

interestingly the same values but i can see it in the images that they are plain (not HDR).

thank you @Honey_Patouceul - yes, you’re right, seems it’s just a warning and did recover after a minute in plain mode. i’ll leave it running for a bit longer…

tried with the clock boost as you suggested and is still running. definitely an improvement, albeit quite worrying in the first place.

now it’s stalled. just on the last camera, though:

[ 779.063168] fence timeout on [ffffffc727695180] after 1500ms
[ 779.063179] name=[nvhost_sync:94], current value=6384 waiting value=6385
[ 779.063185] ---- mlocks ----

[ 779.063218] ---- syncpts ----
[ 779.063232] id 16 (gv11b_511) min 234 max 234 refs 1 (previous client : )
[ 779.063236] id 17 (gv11b_510) min 9 max 9 refs 1 (previous client : )
[ 779.063241] id 18 (gv11b_509) min 17 max 17 refs 1 (previous client : gv11b_509)
[ 779.063246] id 19 (15340000.vic_gst-launch-1.0_0) min 13971 max 13971 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[ 779.063250] id 20 (15340000.vic_gst-launch-1.0_0) min 13923 max 13923 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[ 779.063272] id 21 (progress) min 60537 max 0 refs 1 (previous client : progress)
[ 779.063277] id 22 (stats_progress) min 196744 max 0 refs 1 (previous client : stats_progress)
[ 779.063281] id 24 (gv11b_508_user) min 131072 max 131072 refs 1 (previous client : gv11b_508_user)
[ 779.063287] id 28 (gv11b_507_user) min 131115 max 131094 refs 1 (previous client : gv11b_507_user)
[ 779.063292] id 30 (gv11b_506_user) min 131072 max 131072 refs 1 (previous client : gv11b_506_user)
[ 779.063296] id 32 (gv11b_505_user) min 131072 max 131072 refs 1 (previous client : gv11b_505_user)
[ 779.063300] id 34 (gv11b_504_user) min 131072 max 131072 refs 1 (previous client : gv11b_504_user)
[ 779.063304] id 35 (progress) min 10494 max 0 refs 1 (previous client : progress)
[ 779.063308] id 36 (15340000.vic_nvargus-daemon_0) min 45009 max 45009 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[ 779.063313] id 38 (gv11b_503) min 6 max 6 refs 1 (previous client : gv11b_503)
[ 779.063316] id 39 (gv11b_502) min 8 max 8 refs 1 (previous client : gv11b_502)
[ 779.063320] id 40 (gv11b_501) min 6 max 6 refs 1 (previous client : gv11b_501)
[ 779.063324] id 41 (gv11b_500) min 6 max 6 refs 1 (previous client : gv11b_500)
[ 779.063327] id 42 (gv11b_499) min 6 max 6 refs 1 (previous client : gv11b_499)
[ 779.063331] id 43 (15340000.vic_gst-launch-1.0_0) min 13917 max 13917 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063335] id 44 (15340000.vic_gst-launch-1.0_0) min 10686 max 10686 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063339] id 45 (progress) min 8511 max 0 refs 1 (previous client : progress)
[ 779.063344] id 46 (15340000.vic_nvargus-daemon_0) min 41401 max 41401 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063349] id 48 (gv11b_498) min 6 max 6 refs 1 (previous client : gv11b_498)
[ 779.063352] id 49 (gv11b_495) min 6 max 6 refs 1 (previous client : gv11b_497)
[ 779.063356] id 50 (gv11b_497) min 9 max 9 refs 1 (previous client : gv11b_495)
[ 779.063373] id 51 (gv11b_496) min 6 max 6 refs 1 (previous client : gv11b_496)
[ 779.063377] id 52 (gv11b_494) min 6 max 6 refs 1 (previous client : gv11b_494)
[ 779.063381] id 53 (15340000.vic_gst-launch-1.0_0) min 11310 max 11310 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063385] id 54 (15340000.vic_gst-launch-1.0_0) min 9954 max 9954 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063388] id 55 (progress) min 39515 max 31715 refs 1 (previous client : progress)
[ 779.063392] id 56 (15340000.vic_nvargus-daemon_0) min 7811 max 7811 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063397] id 58 (gv11b_486) min 15 max 15 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063400] id 59 (gv11b_474) min 9 max 9 refs 1 (previous client : gv11b_492)
[ 779.063404] id 60 (gv11b_484) min 12 max 12 refs 1 (previous client : gv11b_491)
[ 779.063425] id 61 (gv11b_487) min 12 max 12 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063429] id 62 (gv11b_488) min 9 max 9 refs 1 (previous client : gv11b_489)
[ 779.063433] id 63 (15340000.vic_gst-launch-1.0_0) min 10665 max 10665 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063438] id 64 (15340000.vic_gst-launch-1.0_0) min 9318 max 9318 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063452] id 65 (15340000.vic_gst-launch-1.0_0) min 2460 max 2460 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063459] id 66 (15340000.vic_nvargus-daemon_0) min 31344 max 31344 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063472] id 67 (15340000.vic_nvargus-daemon_0) min 7113 max 7113 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[ 779.063484] id 68 (gv11b_478) min 9 max 9 refs 1 (previous client : gv11b_488)
[ 779.063495] id 69 (gv11b_485) min 9 max 9 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063502] id 70 (gv11b_481) min 9 max 9 refs 1 (previous client : gv11b_486)
[ 779.063512] id 71 (gv11b_480) min 9 max 9 refs 1 (previous client : gv11b_485)
[ 779.063522] id 72 (gv11b_479) min 6 max 6 refs 1 (previous client : gv11b_484)
[ 779.063533] id 73 (progress) min 14709 max 7605 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063552] id 74 (15340000.vic_gst-launch-1.0_0) min 8823 max 8823 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063557] id 75 (15340000.vic_nvargus-daemon_0) min 6529 max 6529 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063562] id 76 (15340000.vic_gst-launch-1.0_0) min 33208 max 33208 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063566] id 78 (gv11b_483) min 12 max 12 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063570] id 79 (gv11b_482) min 6 max 6 refs 1 (previous client : gv11b_482)
[ 779.063576] id 80 (gv11b_475) min 12 max 12 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063602] id 81 (gv11b_476) min 9 max 9 refs 1 (previous client : gv11b_480)
[ 779.063618] id 82 (gv11b_477) min 6 max 6 refs 1 (previous client : gv11b_479)
[ 779.063628] id 83 (progress) min 14028 max 7608 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063636] id 84 (15340000.vic_gst-launch-1.0_0) min 8160 max 8160 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063641] id 86 (15340000.vic_nvargus-daemon_0) min 34818 max 34818 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063646] id 87 (15340000.vic_nvargus-daemon_0) min 9 max 9 refs 1 (previous client : 15340000.vic_nvargus-daemon_0)
[ 779.063650] id 88 (gv11b_490) min 21 max 21 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063655] id 89 (gv11b_489) min 9 max 9 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063658] id 90 (gv11b_492) min 6 max 6 refs 1 (previous client : gv11b_476)
[ 779.063663] id 91 (gv11b_491) min 6 max 6 refs 1 (previous client : gv11b_475)
[ 779.063667] id 92 (gv11b_493) min 12 max 12 refs 1 (previous client : 15340000.vic_argus_thread_0)
[ 779.063672] id 94 (progress) min 6384 max 1221 refs 1 (previous client : 15340000.vic_argus_thread_0)

[ 779.064057] ---- channels ----
[ 779.064070]
channel 1 - 15340000.vic

[ 779.064078] NvHost basic channel registers:
[ 779.064095] CMDFIFO_STAT_0: 0000003e
[ 779.064108] CMDFIFO_RDATA_0: 00000041
[ 779.064122] CMDP_OFFSET_0: 00000050
[ 779.064126] CMDP_CLASS_0: 00000001
[ 779.064130] CHANNELSTAT_0: 00000000
[ 779.064137]
ffffffc76b105800: JOB, syncpt_id=46, syncpt_val=41404, first_get=000000e0, timeout=10000, num_slots=17
[ 779.064147] GATHER at fe388000+5a44, 3 words
[ 779.064150] 304e0005
[ 779.064153] 0000ec78
[ 779.064156] 00000015

[ 779.064186] GATHER at fe388000+5a50, 29 words
[ 779.064190] 10100002
[ 779.064193] 000001c1
[ 779.064199] 00b90110
[ 779.064203] 10100002
[ 779.064210] 000001c2
[ 779.064222] 1ff7fff0
[ 779.064228] 10100002
[ 779.064239] 000001cc
[ 779.064248] 00000000
[ 779.064255] 10100002
[ 779.064259] 000001c3
[ 779.064261] 1ffcffc0
[ 779.064264] 10100002
[ 779.064267] 000001c8
[ 779.064270] 9ffac000
[ 779.064272] 10100002
[ 779.064275] 000001c9
[ 779.064278] 9ffae200
[ 779.064281] 10100002
[ 779.064284] 00000480
[ 779.064286] 9ffa4000
[ 779.064289] 10100002
[ 779.064293] 00000481
[ 779.064295] 9ffa6800
[ 779.064298] 10100002
[ 779.064301] 000000c0
[ 779.064304] 00000100
[ 779.064309] 20000001
[ 779.064312] 0000042e

[ 779.064318] GATHER at fe388000+5ac4, 2 words
[ 779.064321] 20000001
[ 779.064324] 0000042e

[ 779.064348]
channel 2 - 15820000.se

[ 779.064359] NvHost basic channel registers:
[ 779.064370] CMDFIFO_STAT_0: 00002040
[ 779.064382] CMDFIFO_RDATA_0: 104c5000
[ 779.064392] CMDP_OFFSET_0: 00000000
[ 779.064396] CMDP_CLASS_0: 00000000
[ 779.064400] CHANNELSTAT_0: 00000000
[ 779.064404] The CDMA sync queue is empty.

[ 779.064413]
channel 3 - 15830000.se

[ 779.064425] NvHost basic channel registers:
[ 779.064436] CMDFIFO_STAT_0: 00002040
[ 779.064447] CMDFIFO_RDATA_0: 1080022c
[ 779.064460] CMDP_OFFSET_0: 00000000
[ 779.064464] CMDP_CLASS_0: 00000000
[ 779.064468] CHANNELSTAT_0: 00000000
[ 779.064470] The CDMA sync queue is empty.

[ 779.064477]
channel 4 - 15840000.se

[ 779.064480] NvHost basic channel registers:
[ 779.064483] CMDFIFO_STAT_0: 00002040
[ 779.064487] CMDFIFO_RDATA_0: 40200540
[ 779.064491] CMDP_OFFSET_0: 00000000
[ 779.064495] CMDP_CLASS_0: 00000000
[ 779.064498] CHANNELSTAT_0: 00000000
[ 779.064501] The CDMA sync queue is empty.

[ 779.064509]
---- host general irq ----

[ 779.064513] sync_intc0mask = 0x00000001
[ 779.064516] sync_intmask = 0x50000003
[ 779.064519]
---- host syncpt irq mask ----

[ 779.064522]
---- host syncpt irq status ----

[ 779.064526] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[ 779.064529] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[ 779.064533] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[ 779.064537] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[ 779.064540] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[ 779.064544] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[ 779.064548] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[ 779.064551] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[ 779.064555] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[ 779.064558] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[ 779.064562] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[ 779.064566] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[ 779.064569] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[ 779.064573] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[ 779.064576] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[ 779.064580] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[ 779.064584] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[ 779.064587] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[ 779.064591] syncpt_thresh_cpu0_int_status(18) = 0x00000000
[ 779.064595] syncpt_thresh_cpu0_int_status(19) = 0x00000000
[ 779.064598] syncpt_thresh_cpu0_int_status(20) = 0x00000000
[ 779.064602] syncpt_thresh_cpu0_int_status(21) = 0x00000000
[ 779.091210] tegra194-vi5 15c10000.vi: no reply from camera processor
[ 779.091358] tegra194-vi5 15c10000.vi: vi capture get status failed
[ 780.595177] fence timeout on [ffffffc75f2a2cc0] after 1500ms
[ 780.595191] fence timeout on [ffffffc75f2a2a80] after 1500ms

I wouldn’t be able to further help you, but you may generate some debug data from this that might help experts for further advice.

The trace log may get more information.

Hi Shane,
I am a colleague of Volker. This issue has not been resolved.
Please see the attached trace and nvargus logs captured as per the directions here

I have reduced the frame size as much as possible in our sensor configuration.

This setup works fine with any combination of 5 cameras connected (including two per deserializer/CSI port), but a 6th camera will never start streaming.

6 cameras seem to work at the full frame rate with v4l2-ctl.

Command used to start the cameras:

gst-launch-1.0 -v \
nvarguscamerasrc sensor-id=0 sensor-mode=1 ! \
        'video/x-raw(memory:NVMM),width=(int)2048,height=(int)1280,framerate=(fraction)25/1' ! \
        nvvidconv ! 'video/x-raw,format=(string)I420' ! \
        fpsdisplaysink video-sink=fakesink text-overlay=false \
\
nvarguscamerasrc sensor-id=1 sensor-mode=1 ! \
        'video/x-raw(memory:NVMM),width=(int)2048,height=(int)1280,framerate=(fraction)25/1' ! \
        nvvidconv ! 'video/x-raw,format=(string)I420' ! \
        fpsdisplaysink video-sink=fakesink text-overlay=false \
\
nvarguscamerasrc sensor-id=2 sensor-mode=1 ! \
        'video/x-raw(memory:NVMM),width=(int)2048,height=(int)1280,framerate=(fraction)25/1' ! \
        nvvidconv ! 'video/x-raw,format=(string)I420' ! \
        fpsdisplaysink video-sink=fakesink text-overlay=false \
\
nvarguscamerasrc sensor-id=4 sensor-mode=1 ! \
        'video/x-raw(memory:NVMM),width=(int)2048,height=(int)1280,framerate=(fraction)25/1' ! \
        nvvidconv ! 'video/x-raw,format=(string)I420' ! \
        fpsdisplaysink video-sink=fakesink text-overlay=false \
\
nvarguscamerasrc sensor-id=5 sensor-mode=1 ! \
        'video/x-raw(memory:NVMM),width=(int)2048,height=(int)1280,framerate=(fraction)25/1' ! \
        nvvidconv ! 'video/x-raw,format=(string)I420' ! \
        fpsdisplaysink video-sink=fakesink text-overlay=false \
\
nvarguscamerasrc sensor-id=6 sensor-mode=1 ! \
        'video/x-raw(memory:NVMM),width=(int)2048,height=(int)1280,framerate=(fraction)25/1' ! \
        nvvidconv ! 'video/x-raw,format=(string)I420' ! \
        fpsdisplaysink video-sink=fakesink text-overlay=false

6cam-nvargus-log (9.3 MB)
6cam-trace (2.1 MB)

Please clean the log and get the 6th trace log only for checking.

Hi Shane,
Could you please clarify what you are asking? I’m not sure I understand.
How do I clean the log?
What do you mean by ‘6th trace log’? Do you mean setting the log level to 6 here: echo 2 > /sys/kernel/debug/camrtc/log-level?

Below command for cleaning.

echo > /sys/kernel/debug/tracing/trace

Hello Shane,
That command was executed immediately before running the gst-launch-1.0 command, and the trace was captured after terminating it.
Could you please also explain what you mean by ‘6th trace log’?

Clean all trace log before launch the failed camera and get the trace log after launch failed.

That is what I did. I’ve done that again, please see the attachments.

6cam-nvargus-log-short (16.5 MB)
6cam-trace-short (1.5 MB)

Does your camera board have designed 7 cameras with virtual channel? (sensor-id = 0-6)

Hi Shane,
Our board supports 8 cameras, two per MAX9296 deserialiser with virtual channels. The first trace I sent used cameras 0, 1, 2, 4, 5 and 6, and the second trace used 0, 1, 2, 3, 4 and 5.

Did you review the vc-id and vc_id in your dts file.

Yes, these are all correct.

Each of the cameras works individually; up to 5 will work at the same time.

Can you attached the sensor dts for reviewing.

Attached, as requested.

ar0233-mode-settings.dtsi (5.4 KB)
tegra194-camera-ar0233-a00.dtsi (16.2 KB)