Multi-camera Jetson Xavier NX configuration, more than 6 cameras

Current we are working on it. I can’t tell the time.

hello diego.chaverri,

could you please update the binaries with attachment for verification, Topic184902_Sep22.7z (713.8 KB)
thanks

Hi @JerryChang ,

Thanks for the binaries, it is greatly appreciated.

Can you quickly explain what it is the expected outcome of using those?

What properties are valid for the “position” property of the cameras in the multicapture system?

Regards
Diego

hello diego.chaverri,

it’s a fix to make the sensor placement error harmless, you should follow the path to update these binaries on your target, after that, please perform warm-reboot (i.e. $ sudo reboot) to make the changes take effect.
in the device tree modification, please take the values from 0 to F to assign the positions property.
thanks

Hi @JerryChang ,

Thanks for the information, I could validate that now position assignment for the cameras using 0 thru F is recognized and no longer reports a “No Cameras Available” error.

What is the pattern that nvarguscamerasrc follows in regard to the “sensor-id” property? We have a concern about the order of the cameras when using the “sensor-id” parameter in the pipeline.

Should I expect to have:
Camera 0, position 0 → sensor-id 0
Camera 1, position 1 → sensor-id 1
.
.
.
Camera 10, position A → sensor-id 10
.
.
Camera 15, position F → sensor-id 15

Would this be the expected order?

Thanks

hello diego.chaverri,

that’s an expected orders, could you please also have a try to double confirmation.
thanks

Hi @JerryChang,

While testing the modified binaries that you provided, I have been getting the following error frequently, once it appears, I’m forced to power-cycle the board.

nvidia@nvidia-desktop:~$ DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=1 ! nvvidconv ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 1 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 29,999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:00:02.296718656
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
Setting pipeline to NULL ...
Freeing pipeline ...
  • By inspecting the kernel log, I can see that a timeout is reported:
fence timeout on [ffffffc3eba5c3c0] after 1500ms
  • I found the following entry for a developer having the same issue while working with JetPack4.5:

https://forums.developer.nvidia.com/t/errors-starting-6-nvarguscamerasrc-pipelines-in-r32-5-0/170477

  • Apparently, the fix is a modification also to one of the nvarguscamera-related libraries. Would it be possible for you to generate a modified version for L4T32.6.1 (JP4.6) of the libgstnvarguscamerasrc.so with the increment in the timeout as suggested in the post?

Thanks

hello diego.chaverri,

please try the attachment for verification, Topic184902_Sep27_libgstnvarguscamerasrc.7z (24.2 KB)
thanks

Hi @JerryChang

Thanks for the binaries, the test I’m running involves both linear and HDR (we have enabled PWL HDR) capture modes.

  • In the case of the linear, the capture seems to be stable now, I have not gotten any timeout errors while using the provided libgstnvarguscamerasrc binary.

  • In the case of HDR, the capture still crashes with the aforementioned error. Is there any other consideration from the nvarguscamerasrc point of view when working with HDR? Does the timeout value need additional adjustment?

Thanks

hello diego.chaverri,

what’s the error messages shown, could you please enable VI tracing log to dump the details.
is this failure related to luminance changes? for example, are you target the camera facing the bright scene.

Hi @JerryChang,

The error as I execute the pipeline is the same as described above:

nvidia@nvidia-desktop:~$ DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=1 ! nvvidconv ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 1 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 29,999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:00:02.296718656
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
Setting pipeline to NULL ...
Freeing pipeline ...

The Kernel log information that is generated once the pipeline fails:

[55056.009350] fence timeout on [ffffffc35246e3c0] after 1500ms
[55056.009357] tegra194-vi5 15c10000.vi: no reply from camera processor
[55056.009378] tegra194-vi5 15c10000.vi: vi capture get status failed
[55056.009659] name=[nvhost_sync:38], current value=0 waiting value=1
[55056.009667] ---- mlocks ----

[55056.009716] ---- syncpts ----
[55056.009729] id 2 (disp_a) min 550 max 550 refs 1 (previous client : )
[55056.009736] id 3 (disp_b) min 2 max 2 refs 1 (previous client : )
[55056.009750] id 8 (vblank0) min 263219 max -4 refs 1 (previous client : )
[55056.009769] id 19 (gv11b_511) min 94390 max 94390 refs 1 (previous client : )
[55056.009803] id 20 (gv11b_510) min 14 max 14 refs 1 (previous client : )
[55056.009811] id 21 (gv11b_509) min 41769 max 41769 refs 1 (previous client : gv11b_509)
[55056.009819] id 22 (15340000.vic_gst-launch-1.0_0) min 30000 max 30000 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[55056.009834] id 30 (gv11b_507_user) min 23 max 0 refs 1 (previous client : )
[55056.009853] id 37 (15340000.vic_gst-launch-1.0_0) min 30000 max 30000 refs 1 (previous client : 15340000.vic_gst-launch-1.0_0)
[55056.009865] id 42 (gv11b_501) min 604 max 604 refs 1 (previous client : gv11b_503)
[55056.009871] id 43 (gv11b_499) min 603 max 603 refs 1 (previous client : gv11b_502)
[55056.009878] id 44 (gv11b_502) min 604 max 604 refs 1 (previous client : gv11b_500)
[55056.009885] id 45 (gv11b_503) min 603 max 603 refs 1 (previous client : gv11b_499)
[55056.009891] id 46 (gv11b_500) min 603 max 603 refs 1 (previous client : gv11b_501)

[55056.010757] ---- channels ----
[55056.010796] 
               channel 2 - 15820000.se

[55056.010802] NvHost basic channel registers:
[55056.010810] CMDFIFO_STAT_0:  00002040
[55056.010817] CMDFIFO_RDATA_0: 01013040
[55056.010827] CMDP_OFFSET_0:   00000000
[55056.010834] CMDP_CLASS_0:    00000000
[55056.010840] CHANNELSTAT_0:   00000000
[55056.010846] The CDMA sync queue is empty.

[55056.010861] 
               channel 3 - 15830000.se

[55056.010866] NvHost basic channel registers:
[55056.010872] CMDFIFO_STAT_0:  00002040
[55056.010878] CMDFIFO_RDATA_0: b0800188
[55056.010887] CMDP_OFFSET_0:   00000000
[55056.010893] CMDP_CLASS_0:    00000000
[55056.010898] CHANNELSTAT_0:   00000000
[55056.010903] The CDMA sync queue is empty.

[55056.010914] 
               channel 4 - 15840000.se

[55056.010918] NvHost basic channel registers:
[55056.010924] CMDFIFO_STAT_0:  00002040
[55056.010930] CMDFIFO_RDATA_0: 42000068
[55056.010937] CMDP_OFFSET_0:   00000000
[55056.010942] CMDP_CLASS_0:    00000000
[55056.010948] CHANNELSTAT_0:   00000000
[55056.010953] The CDMA sync queue is empty.

[55056.010992] 
               ---- host general irq ----

[55056.010999] sync_intc0mask = 0x00000001
[55056.011005] sync_intmask = 0x50000003
[55056.011009] 
               ---- host syncpt irq mask ----

[55056.011014] 
               ---- host syncpt irq status ----

[55056.011021] syncpt_thresh_cpu0_int_status(0) = 0x00000000
[55056.011027] syncpt_thresh_cpu0_int_status(1) = 0x00000000
[55056.011033] syncpt_thresh_cpu0_int_status(2) = 0x00000000
[55056.011039] syncpt_thresh_cpu0_int_status(3) = 0x00000000
[55056.011045] syncpt_thresh_cpu0_int_status(4) = 0x00000000
[55056.011051] syncpt_thresh_cpu0_int_status(5) = 0x00000000
[55056.011371] syncpt_thresh_cpu0_int_status(6) = 0x00000000
[55056.011379] syncpt_thresh_cpu0_int_status(7) = 0x00000000
[55056.011386] syncpt_thresh_cpu0_int_status(8) = 0x00000000
[55056.011392] syncpt_thresh_cpu0_int_status(9) = 0x00000000
[55056.011398] syncpt_thresh_cpu0_int_status(10) = 0x00000000
[55056.011404] syncpt_thresh_cpu0_int_status(11) = 0x00000000
[55056.011437] syncpt_thresh_cpu0_int_status(12) = 0x00000000
[55056.011443] syncpt_thresh_cpu0_int_status(13) = 0x00000000
[55056.011449] syncpt_thresh_cpu0_int_status(14) = 0x00000000
[55056.011454] syncpt_thresh_cpu0_int_status(15) = 0x00000000
[55056.011460] syncpt_thresh_cpu0_int_status(16) = 0x00000000
[55056.011466] syncpt_thresh_cpu0_int_status(17) = 0x00000000
[55056.011472] syncpt_thresh_cpu0_int_status(18) = 0x00000000
[55056.011478] syncpt_thresh_cpu0_int_status(19) = 0x00000000
[55056.011580] syncpt_thresh_cpu0_int_status(20) = 0x00000000
[55056.011588] syncpt_thresh_cpu0_int_status(21) = 0x00000000
[55056.221699] tegra-i2c 3180000.i2c: no acknowledge from address 0x21
[55056.222016] tegra-i2c 3180000.i2c: no acknowledge from address 0x21
[55056.413702] tegra-i2c 3180000.i2c: no acknowledge from address 0x11
[55056.857630] tegra-i2c 3180000.i2c: no acknowledge from address 0x21
[55056.857939] tegra-i2c 3180000.i2c: no acknowledge from address 0x21
[55057.497578] tegra-i2c 3180000.i2c: no acknowledge from address 0x21
[55057.497868] tegra-i2c 3180000.i2c: no acknowledge from address 0x21

As you can see, the error is exactly the same as the previous one. The modified binary that you provided completely improved the behavior of the linear mode capture, however, HDR constantly fails, that’s the reason why I was asking if there is any other timing consideration when working with HDR.

is this failure related to luminance changes? for example, are you target the camera facing the bright scene.

It is not, it fails independently of any lighting condition, whenever I try to capture using HDR (sensor-mode=1), the error occurs. Let me know if you need any additional information.

Thank you

@JerryChang,

Also attaching the trace as you requested, the previous reply exceeded the maximum message body limit.

I enabled the following traces:

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace

The log for the trace is attached.

ThanksVITrace.txt (199.5 KB)

hello diego.chaverri,

is this HDR mode ever verified with previous release?
there’s an error to report PIXEL_SHORT_LINE, the line-length is 941 according to the register dumps,
could you please review sensor active region settings in the device tree. thanks

Hi @JerryChang,

I can confirm that the HDR capture was working in JetPack4.4. Just to summarize so far what I’m seeing using your modified binaries:

STOCK BINARIES:

  • Linear mode: Very unstable, constantly fails with the dmabuf_fd -1 mapped entry NOT found error.
  • HDR mode: Same as with linear.

MODIFIED BINARIES

  • Linear mode: Very stable, I have NOT seen the dmabuf_fd -1 mapped entry NOT found error for a while now, through numerous iterations of playing / stop the pipeline cycles.
  • HDR mode: Very unstable, the success rate for the HDR capture is lower than when using the stock libraries. Out of 10 attempts to actually capture in HDR mode, the pipeline will work 1-2 times.

Could you please review sensor active region settings in the device tree

The active width an active height match the sensor resolution as specified in the technical documentation. One parameter that it is still unclear to me how is it supposed to be calculated is the line_length. From a previous JetPack reference, for the line_length:

Specifies the pixel line width horizontal timing size for the sensor mode for calibrating the features in the camera stack.
This value must be equal or larger than active_w.

Any additional hint that you can provide it is greatly appreciated as having stable HDR capture is critical for the customer.

Thanks

hello diego.chaverri,

line_length is the setting according to sensor init register configuration, it’s a timing value.
however, your driver settings should be verified since they’ve confirmed on JP-4.4.

may I also know what’s the HDR mode type, for example, is it PWL-WDR or DOL-WDR?
thanks

Hi @JerryChang,

May I also know what’s the HDR mode type, for example, is it PWL-WDR or DOL-WDR?

We are using PWL-WDR for the HDR mode type.

Thanks

hello diego.chaverri,

had you also try with other plugins?
for example, what’s the test results with nv3dsink, nvoverlaysink, or even, fakesink?

Hi @JerryChang,

had you also try with other plugins?
for example, what’s the test results with nv3dsink , nvoverlaysink , or even, fakesink ?

Yes, I can confirm that even using the least error-prone case which would be fakesink, the pipeline fails exactly the same when trying to capture in HDR mode.

nvidia@nvidia-desktop:~$ DISPLAY=:0 gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=1 ! fakesink silent=false -v
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event   ******* (fakesink0:sink) E (type: stream-start (10254), GstEventStreamStart, stream-id=(string)10d9d5cfb24ef2bc27e142a7306cdcc1, flags=(GstStreamFlags)GST_STREAM_FLAG_NONE, group-id=(uint)1;) 0x55748a01c0
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFakeSink:fakesink0: last-message = event   ******* (fakesink0:sink) E (type: caps (12814), GstEventCaps, caps=(GstCaps)"video/x-raw\(memory:NVMM\)\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ format\=\(string\)NV12\,\ framerate\=\(fraction\)30/1";) 0x55748a0230
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: 3840 x 2160 FR = 29,999999 fps Duration = 33333334 ; Analog Gain range min 1,000000, max 16,000000; Exposure Range min 100000, max 22000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 1 
   Output Stream W = 3840 H = 2160 
   seconds to Run    = 0 
   Frame Rate = 29,999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
Execution ended after 0:00:02.292318012
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
Setting pipeline to NULL ...
Freeing pipeline ...

Let me know if there is any other test it is worth trying.

Regards

hello diego.chaverri,

could you please do some further testing. thanks!

  1. could you please have a try to restart argus_daemon every-time before enable nvarguscamerasrc.

  2. please try add enableCamInfiniteTimeout=1 before running nvargus-daemon.
    for example,
    sudo pkill nvargus-daemon
    export enableCamInfiniteTimeout=1
    sudo nvargus-daemon

  3. how about v4l2 standard utility, did you still see the same failure on PWL-WDR mode?
    i.e. v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.