Jetson Thor: argus_camera fails to stream 14 cameras simultaneously

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Hi NV Team,

We are developing a camera driver on the Jetson Thor platform, supporting a total of 14 cameras with the following configurations:

  • 3840 x 2160 (8MP): 10 channels
  • 1920 x 1536 (3MP): 4 channels

Here are the details of our test results and the issue we encountered:

1. V4L2 Raw Streaming (Success)

We verified the hardware and low-level drivers using v4l2-ctl. All 14 cameras can stream simultaneously and achieve the target frame rates without issues. The commands used are as follows:

  • For 8MP cameras:
    v4l2-ctl -V --set-fmt-video=width=3840,height=2160 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000000 -d /dev/video*

  • For 3MP cameras:
    v4l2-ctl -V --set-fmt-video=width=1920,height=1536 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10000000 -d /dev/video*

2. Libargus / argus_camera (Failure)

When switching to argus_camera, we cannot light up all 14 cameras simultaneously.

  • We can successfully stream up to 11 cameras at the same time.
  • The moment we attempt to open the 12th camera, the video streams for all cameras freeze completely.

3. Attempted Fixes

We found a similar topic on the forum and applied the suggested modifications, but the issue persists: Nvvidconv slow in multi-process: why does performance collapse when using multiple GStreamer processes? - #6 by DaneLLL

image
image1467×519 23.2 KB

Questions:

  1. ISP Load Balancing: Jetson Thor features two ISP blocks. Does the system automatically perform load balancing across these two ISPs?
  2. Device Tree Configuration: If load balancing is not automatic, how should we configure the Device Tree (dts) to evenly distribute the 14 camera channels between the two ISPs to avoid hitting hardware/bandwidth limits?
  3. Argus Limits: Is there any hardcoded limitation or resource bottleneck within nvargus-daemon for managing more than 11 concurrent streams on Jetson Thor?

Any guidance, patches, or reference dts configurations would be highly appreciated.

Thanks!

4

hello norman_xu,

may I know what capture mode you’ve enabled for testing? for instance, is it multiSession mode?

it’s v4l2 standard IOCTL to verify basic camera stream. however, it doesn’t evolve ISP and camera operations.
please also execute through gst pipeline with nvarguscamerasrc plugin.
you may disable preview and shows frame-rate only for quick testing.
for instance,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

5

Hi @JerryChang ,

  1. Yes, we are using the multiSession mode for testing.

  2. We executed the test using the suggested gst-launch-1.0 pipeline with nvarguscamerasrc (disabling preview and enabling fpsdisplaysink to monitor framerate).

gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=3840, height=2160, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

Result: We observed a bottleneck at 11 cameras. The streaming works fine with up to 11 instances running simultaneously. However, the moment we attempt to launch the 12th camera, all active cameras stop printing framerates and the entire pipeline hangs/fails.

/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 10635, dropped: 0, current: 25.07, average: 28.43
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 10648, dropped: 0, current: 23.75, average: 28.42
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 10660, dropped: 0, current: 22.82, average: 28.42
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 10673, dropped: 0, current: 24.47, average: 28.41
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 10686, dropped: 0, current: 24.29, average: 28.40
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 10694, dropped: 0, current: 10.59, average: 28.37
(Argus) Error 0x00000005:  (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error 0x00000005:  (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 92)
CONSUMER: Done Success
Caught SIGSEGV:99:99.
Spinning.  Please run 'gdb gst-launch-1.0 6457' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.
Got EOS from element "pipeline0".
Execution ended after 0:06:33.589928812
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up

image
image1920×1032 102 KB

6

hello norman_xu,

it should support up-to 16 cameras, ideally.
could you please have below commands to dump device tree into dts formats for quick checking.
for instance, $ sudo dtc -I fs -O dts /sys/firmware/devicetree/base > /tmp/123.dts

7

Hi @JerryChang ,

Attached is the device tree file

123.dts.txt (407.9 KB)

8

hello norman_xu,

BTW, do you still working with JP-7.1/r38.4?
since JetPack 7.2 is now available, if that’s possible, please moving forward to the latest version for verification.

9

Hi @JerryChang ,

We are currently developing a camera driver on the JetPack 7.2 platform using the SIPL framework.

However, we noticed that we are missing the corresponding JSON configuration files for “/usr/lib/nvsipl_drv/libnvsipl_qry*.so”.

Could you please provide a reference camera json file for our development?

Unable to locate SIPL JSON files on JetPack 7.2 - Jetson Systems / Jetson Thor - NVIDIA Developer Forums

10

hello norman_xu,

unfortunately, we don’t have a reference camera JSON file at the moment.
please follow developer guide to make it for your use-case, SIPL Query JSON Guide — NVIDIA Jetson Linux Developer Guide