Jetson Orin NX: 5-camera design with GMSL + MIPI CSI-2, ISP, encode and decode capability

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
1

Hello NVIDIA team,

I am evaluating Jetson Orin NX for a 5-camera edge AI system, and I would like to confirm the best architecture and the practical performance limits for camera capture, ISP, encode, and decode.

My system has 5 cameras running simultaneously. I am considering the following two hardware options:

Option 1:

  • 4 cameras use GMSL with serializers and deserializer.

  • The GMSL deserializer works in Virtual Channel / VS mode.

  • These 4 cameras are connected to Jetson Orin NX through one 4-lane MIPI CSI-2 interface.

  • The 5th camera is connected separately through another 4-lane MIPI CSI-2 interface.

  • All 5 cameras output RAW Bayer data and require ISP processing on Jetson Orin NX.

Option 2:

  • 4 cameras still use GMSL with serializers and deserializer.

  • The 5th camera uses an external ISP first.

  • The 5th camera output after external ISP is then connected to Jetson Orin NX through a 4-lane MIPI CSI-2 interface.

  • In this case, only the 4 GMSL cameras require Jetson Orin NX internal ISP processing, while the 5th camera is already ISP-processed externally.

I would like to ask the following questions:

  1. Camera capture and MIPI CSI-2

    • Does Jetson Orin NX support this kind of 5-camera architecture?

    • For Option 1, can 4 GMSL cameras be sent through one 4-lane MIPI CSI-2 interface using Virtual Channel / VS mode, while the 5th camera uses another 4-lane MIPI CSI-2 interface?

    • Are there any limitations related to NVCSI pixel parsers, virtual channels, or CSI bandwidth that I should consider for 5 simultaneous cameras?

  2. Hardware ISP

    • How many camera streams can the Jetson Orin NX internal ISP process simultaneously?

    • Can Jetson Orin NX process ISP for 5 RAW Bayer camera streams at the same time?

    • If not, would Option 2 be a better design, where only 4 GMSL RAW cameras use the internal ISP and the 5th camera uses an external ISP?

    • What is the maximum practical resolution and frame rate for 4 or 5 simultaneous ISP-processed camera streams?

  3. Hardware video encoding

    • If all 5 cameras run simultaneously, what is the maximum supported hardware encode resolution and frame rate?

    • Can Jetson Orin NX encode 5 camera streams at 1080p@30fps, 1080p@60fps, or 4K@30fps?

    • What are the practical limits for H.264 and H.265 encoding when 5 camera streams are active?

    • If internal ISP, AI inference, and encoding are running at the same time, should we expect lower encode performance than the values listed in the datasheet?

  4. Hardware video decoding

    • If the system also needs video decoding at the same time, what is the maximum decode capability while 5 cameras are active?

    • Can Jetson Orin NX decode multiple H.265 or H.264 streams while simultaneously capturing, ISP-processing, inferencing, and encoding camera streams?

    • Are NVDEC and NVENC independent enough to run concurrently with the 5-camera pipeline?

  5. End-to-end pipeline

    • For a real product with 5 cameras + GMSL + MIPI CSI-2 + ISP + AI inference + optional encoding/decoding, which architecture would NVIDIA recommend: Option 1 or Option 2?

    • Are there any reference designs, device-tree examples, GStreamer pipelines, or performance benchmarks for a 5-camera GMSL system on Jetson Orin NX?

    • Are there specific GMSL deserializers or camera modules that NVIDIA recommends for this type of design?

From the Jetson Orin NX datasheet, I understand that the ISP throughput is limited, NVCSI supports multiple streams, and NVENC/NVDEC limits depend on codec and resolution. However, I would like to confirm the practical supported configuration for a real 5-camera product.

Thank you.

2

— 🤖 This is an automated AI response. I’m here to help, but please verify important details! —

Based on the provided context, I will answer the questions related to the Jetson Orin Nano and Jetson Orin NX.

  1. Camera capture and MIPI CSI-2:
    The Jetson Orin NX supports up to 16 virtual channels, but it may have limitations related to NVCSI pixel parsers, virtual channels, or CSI bandwidth for 5 simultaneous cameras. According to the context, a single CSI brick can support x4 VC streams, and the Orin Module brings eight MIPI CSI lanes to the connector, supporting up to two quad-lane camera streams or up to four dual-lane camera streams.

  2. Hardware ISP:
    The context does not explicitly state the maximum number of camera streams that the Jetson Orin NX internal ISP can process simultaneously. However, it mentions that the ISP throughput is limited, and NVCSI supports multiple streams.

  3. Hardware video encoding:
    The maximum supported hardware encode resolution and frame rate for 5 simultaneous camera streams are not explicitly stated in the context. However, it mentions that the NVENC limits depend on the codec and resolution.

  4. Hardware video decoding:
    The context does not provide explicit information on the maximum decode capability while 5 cameras are active. However, it mentions that NVDEC and NVENC are independent enough to run concurrently with the 5-camera pipeline.

  5. End-to-end pipeline:
    The context does not provide a clear recommendation for the architecture (Option 1 or Option 2) for a real product with 5 cameras + GMSL + MIPI CSI-2 + ISP + AI inference + optional encoding/decoding. However, it suggests that the Jetson Orin NX supports various camera configurations and provides a platform for building and deploying AI-powered applications.

In summary, while the context provides some information on the Jetson Orin NX’s capabilities, it does not fully address the questions related to the practical performance limits for camera capture, ISP, encode, and decode for a 5-camera system. Therefore, I could not find clear support for some parts of the question in the provided Jetson Orin Nano documentation.

— We support team will also look into your issue to have further suggestions soon. please stay tuned. —

5

MIPI CSI-2 Cameras: Supports up to 4 cameras (two 4-lane or four 2-lane configurations). By utilizing Virtual Channels (VC), this can be expanded up to 8 streams.

5 cameras can support without problem for both option.

BTW, you need design your own carrier board instead of using devkit.

Current devkit only support 2 lanes CAM0 and 4 lanes CAM1 connector.

Thanks

7

I am using IMX900 camera sensor with max9295 serializer and max96724 deserializer but in the nvidia bsp package max96724 deserializer driver is not located.

Do you have any reference driver for this or someone posted on this nvidia community forum which helps me with ?

Also IMX900 driver is no longer present in the nvidia bsp package, could you please help me with any reference driver or available driver in this discussion forum.

It means a lot, if this helpful.
Thanks.

8

We don’t have them. You need follow the programing guide to implement it.

Thanks