Why 1 of the streams is terminated when initiating 9 Deepstream pipelines in parallel in Jetpack 6.0

[kaiyit.kok]

Hi, I have encountered an issue when running simple pipelines in parallel which consists of video source, video decoder, and fakesink (sync=false) only. When running 8 or below pipelines in parallel, all the pipelines can be executed without issue, but when running with 9 onwards, at least 1 of the pipelines will be killed during the initialization.

The jetson clocks is enabled and the power mode is set to 0. The video source resolution is 3840x2160, h265 format, 30fps.

I’m running with Jetpack 6.0 and Deepstream 7.0 on Orin NX. Same issue occurs when running on Orin AGX with the same software version. However, this issue does not occur on Orin AGX when running with Jetpack 5.0.2 and Deepstream 6.1.

Is there any configuration that can be done to avoid this issue with Jetpack 6.0 and Deepstream 7.0?

Logs:

Kernel Logs:

[yuweiw]

You can refer to our View Jetson Orin Technical Specifications, we can only support 7x 4K30 (H.265) for Jetson AGX Orin 64GB. If the video source exceeds this number, there may be an exception

[kaiyit.kok]

Does the data in the table really mean this? If it is, my Orin AGX 64GB decoder with Jetpack 5.0.2 can process at least 12 video streams in parallel without issue, and capable to have10 streams in parallel for maintaining 30fps with video resolution 3840x2160:

And the provided performance by Nvidia for Orin AGX 64GB:

And what more interesting is, there is no issue to run with 12 streams in Jetpack 5.0.2 version. But running in the same hardware with Jetpack 6.0, we can only run up to 8 streams to avoid the mentioned issue of this discussion.

Do you really think this is normal and we can refer to the data table for this matter? Can’t we have someone from your side to test this out for the Jetpack 5.0.2 and Jetpack 6.0.

If the issue happens in Jetpack 5.0.2 but not in 6.0, it will be great and expected because newer version always meant to be better and less bugs. However, current issue shows that newer version is worse than older version in handling the decoding.

Hence, can I have a better answer for this matter, other than referring to the data table?

[DaneLLL]

Hi,
Please refer to capability in module data sheet. This is the spec of each Jetpack release. Jetpack 5.0.2 is first Jetpack 5 release and may not be stable. Later releases are stable and fit the spec.

[kaiyit.kok]

Hi DaneLLL, I would love to refer to the table if possible, but I’m encountering the same issue for Orin NX, regardless video with 3840x2160 or 1920x1080 resolution, the maximum stream number can only up to 8 just like the Orin AGX. If referring to the table, it should be up to 18 streams for 1920x1080 resolution with 30fps.

[DaneLLL]

Hi,
Please check
InitNVDEC: Host1x channel open failed - #12 by DaneLLL

This is considered to be constraint of Jetpack 6. Please run 18 threads for decoding 1080p in single process.

[kaiyit.kok]

Hi DaneLLL, I have tried to use multi-threads to run each process, but the same issue still occurs. Here’s the sample:

def run_subprocess(command, thread_id, log_file_full_path):
----with open(log_file_full_path, “w”) as log_file:
--------profile_service = subprocess.Popen(command, stderr=log_file, stdout=log_file, shell=True)

for index, command in enumerate(self.workload_list):
----thread = threading.Thread(target=run_subprocess, args=(command, index, log_file_full_path))
----threads.append(thread)
----thread.start()

for thread in threads:
----thread.join()

[DaneLLL]

Hi,
Please try gst-launch-1.0 command like:

$ gst-launch-1.0 filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink filesrc location=test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! fakesink

We can run the command successfully. It should be good to have 12 decoding threads in single process.

[yingliu]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks