Why 1 of the streams is terminated when initiating 9 Deepstream pipelines in parallel

Hi, I have encountered an issue when running simple pipelines in parallel which consists of video source, video decoder, and fakesink (sync=false) only. When running 8 or below pipelines in parallel, all the pipelines can be executed without issue, but when running with 9 onwards, at least 1 of the pipelines will be killed during the initialization.

The video source resolution is 3840x2160. I’m running with Jetpack 6.0 and Deepstream 7.0. Same issue occurs when running on Orin AGX with the same software version. However, this issue does not occur on Orin AGX when running with Jetpack 5.0.2 and Deepstream 6.1.

Is there any configuration that can be done to avoid this issue with Jetpack 6.0 and Deepstream 7.0?

Have you enabled the max power as Performance — DeepStream documentation 6.4 documentation?

What is the video codec format? H264 or H265? What is the frame rate of the video?
According to Jetson specification, AGX Orin supports 7x 4k@30fps H265 decoding.

Hi Fiona, video codec format = H265 and frame rate = 30fps. Yes, I have enabled jetson clock and power mode is set to 0.

It is okay if the pipeline FPS drop below 30fps when more streams are running in parallel which is what happened in Orin AGX with Jetpack 5.0.2 and Deepstream 6.1.

The current issue is that when Jetpack 6.0 and Deepstream 7.0 is used, regardless Orin NX or Orin AGX, 1 or more streams will be terminated when the number of pipelines is more than 8 during the pipeline initialization stage.

What I’m expecting is that the pipeline FPS can drop below 30 when more pipelines are executed in parallel without terminating any pipeline in the execution. May I know what is happening in this issue? Is there any ways to resolve this issue?

Can you capture the kernel log when you run this case?

This is the latest kernel log on Jetson Orin NX. When I start the execution, the kernel log actually didn’t have new message throughout the execution. Am I missing any configurations of kernel logging?

Current kernel logging configuration:
APPEND ${cbootargs} root=PARTUUID=1ebe4fb5-6b80-4047-8a78-238b5bdb29ed rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nospectre_bhb video=efifb:off console=tty0 nv-auto-config

cat /proc/sys/kernel/printk: 6 6 1 7

Can you provide the complete log which captured right after the process being killed?

Setting pipeline to PAUSED …
Opening in BLOCKING MODE
Pipeline is PREROLLING …
/GstPipeline:pipeline0/GstH265Parse:h265parse0.GstPad:src: caps = video/x-h265, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:4:0:0, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, profile=(string)main, tier=(string)main, level=(string)4
NvMMLiteOpen : Block : BlockType = 279
InitNVDEC: Host1x channel open failed
NVMEDIA: NvMMDecNvVideoCreateParser: 7930: - Failed to get NVDEC Channel handle
NvMMLiteBlockCreate : Block : BlockType = 279
/GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0.GstPad:sink: caps = video/x-h265, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, colorimetry=(string)1:4:0:0, parsed=(boolean)true, stream-format=(string)byte-stream, alignment=(string)au, pixel-aspect-ratio=(fraction)1/1, profile=(string)main, tier=(string)main, level=(string)4
NVDEC_COMMON: Host1x channel open failed
******tegraH265DecoderCreate 1562 Failed in the API InitNVDEC_safe, calling tegraH265DecoderDestroy ******
NVMMLITE_NVVIDEODEC, <cbBeginSequence, 1776> ErrorInfo = VideoErrorInfo_NvVideoDecoderCreate cctx = 0x880686b0
NVMMLITE_NVVIDEODEC, NvVideoBufferProcessing:6596 video_parser_parse Unsupported Codec
Stream format not found, dropping the frame
Stream format not found, dropping the frame
NVMMLITE_NVVIDEODEC, NvMMLiteNvVideoDecDoWork:7100 NVVIDEO Video Dec Unsupported Stream
NVMMLITE_NVVIDEODEC, NvVideoBufferProcessing:6596 video_parser_parse Unsupported Codec
NVMMLITE_NVVIDEODEC, NvMMLiteNvVideoDecDoWork:7100 NVVIDEO Video Dec Unsupported Stream
NVMMLITE_NVVIDEODEC, NvVideoBufferProcessing:6596 video_parser_parse Unsupported Codec
ERROR: from element /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0: Failed to process frame.
Additional debug info:
/dvs/git/dirty/git-master_linux/3rdparty/gst/gst-v4l2/gst-v4l2/gstv4l2videodec.c(2279): gst_v4l2_video_dec_handle_frame (): /GstPipeline:pipeline0/nvv4l2decoder:nvv4l2decoder0:
Maybe be due to not enough memory or failing driver
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to NULL …
ERROR: from element /GstPipeline:pipeline0/GstH265Parse:h265parse0: Internal data stream error.
Additional debug info:
…/libs/gst/base/gstbaseparse.c(3681): gst_base_parse_loop (): /GstPipeline:pipeline0/GstH265Parse:h265parse0:
streaming stopped, reason error (-5)
ERROR: pipeline doesn’t want to preroll.
free(): double free detected in tcache 2

The complete kernel log is needed.

This is from Orin AGX:

Can you provide the complete kernel log before and after the NVDEC error occurs?

mostly get warning, except tat VMData exceed limit, but there is no error before and after the NVDEC error occurs. Maybe i’m missing some configurations or commands for more detailed logging, can you provide the instruction steps if possible?

According to Jetson specification, AGX Orin supports 7x 4k@30fps H265 decoding. The case with more sources than the spec is not guaranteed.

now is not about how many streams are supported for 4kx30fps h265 decoding. it is about why there are stream/s been terminated when exceeding certain limit, which doesn’t happen in Jetpack 5.0.2 but in Jetpack 6.0. what is the solution to avoid this? based on my understanding, the established decoder performance is only stating the maximum number of streams to maintain the target fps, which is completely not the case right now.

Plesae refer to Jetson Modules, Support, Ecosystem, and Lineup | NVIDIA Developer


The red circle is the h265 decoder spec for AGX Orin. This is the hardware limitation.

Perhaps you are not getting it. Here’s the case, Jetson Orin NX and Orin AGX with Jetpack 6.0 have this issue, but Orin AGX with Jetpack 5.0.2 does not have this issue.

It can be narrowed down by referring the Orin AGX, Jetpack 5.0.2 does not have this issue but Jetpack 6.0 have this issue.

It is pointless for showing the platform performance data, because right now I’m talking about the same platform but different behaviour when using different Jetpack version.

The case out of the spec is not guranteed.

Then how can we configure to update limits or use boot option ignore_rlimit_data in Jetson Orin NX or AGX based on the error message?

This is the request for the JetPack. You may raise your request in Latest Jetson & Embedded Systems/Jetson AGX Orin topics - NVIDIA Developer Forums forum.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks