Jetson AGX Orin unexpectedly reboots while running DeepStream + Triton containers

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hardware / Software Environment

Hardware Platform

  • Jetson AGX Orin 64 GB Developer Kit

DeepStream Version

  • DeepStream 7.1

  • Docker image: nvcr.io/nvidia/deepstream-l4t:7.1-triton-multiarch

JetPack Version

  • JetPack 6.2.1

  • L4T 36.4.4

System Setup

We are developing a visual inspection application for Jetson devices using DeepStream.

Our setup consists of two Docker containers running on the same Jetson AGX Orin:

  • A DeepStream 7.1 container running the video processing pipeline.

  • A dedicated Triton Inference Server container (nvcr.io/nvidia/tritonserver:24.02-py3-igpu) hosting all inference models.

The DeepStream pipeline communicates with the Triton server for inference.

Problem Description

Hi all,

after running for some time (sometimes after a few hours), the Jetson unexpectedly reboots without any warning.

The reboot appears to be a full system restart rather than an application or container crash.

Running last reboot shows multiple unexpected reboots:

jetson@jetson-dev-01:~$ last reboot

reboot   system boot  5.15.148-tegra   Mon Jul  6 12:20   still running
reboot   system boot  5.15.148-tegra   Mon Jul  6 09:17   still running
reboot   system boot  5.15.148-tegra   Mon Jul  6 06:12   still running
reboot   system boot  5.15.148-tegra   Sun Jul  5 05:49   still running
reboot   system boot  5.15.148-tegra   Sun Jul  5 04:25   still running
reboot   system boot  5.15.148-tegra   Sat Jul  4 23:35   still running
...

Observations

While the application is running, we continuously monitor the system using jtop.

We have not observed any obvious indicators that would explain the reboot:

  • No abnormal GPU or CPU temperatures.

  • No increasing memory usage or memory leak.

  • No excessive CPU utilization.

  • No significant resource spikes before the reboot.

  • No fatal errors in the application logs.

  • No Docker container crashes before the system restarts.

From the application’s perspective, the reboot appears completely unexpected.

Questions

  1. Could this be related to running JetPack 6.2.1 (L4T 36.4.4) instead of L4T 36.4, which is listed as the officially supported version for DeepStream 7.1?

  2. What would be the recommended approach to identify the root cause of these unexpected system reboots?

  3. Is connecting a serial console currently the best (or only) practical way to capture the information needed to diagnose this type of system-level crash?

Any suggestions on how to further investigate this issue would be greatly appreciated.

Thank you!

2

Hello @Chriwo!

Based on the title and content of your topic, it looks like it may receive better visibility and feedback in a different category. We took the liberty of moving it for you.

If this was an incorrect assessment, please send me a direct message.

Disclaimer: this moderation suggestion and message were generated with AI assistance.

5
  1. could you share the complete media pipeline in the DeepStream application? As shown in the compatibility table, DS7.1 requires 6.1 GA. please use the right version.
  2. Please collect the Jetson bug report after the reboot happens.
    sudo /usr/bin/nvidia-bug-report-jetson.sh, This will generate a system log file. If the script can’t be found, you can also use other cmd, for exmaple,
journalctl --list-boots
journalctl -b -1 -k
journalctl -b -1 -p err
  1. If the reboot is caused by a kernel panic, watchdog reset, power issue, or other hard reset. In those cases, the last messages may not be written to disk, so journalctl/dmesg may not show the root cause after the board comes back up. Connecting a serial console will be practical way to capture the information needed .
6

Hi, thanks for your reply, and apologies for the delayed response on my side.

We downgraded to JP 6.1, but unfortunately the unexpected restarts were still occurring. We also collected logs using journal and the nvidia-bug-report-jetson.sh script, but they didn’t provide any useful clues about the root cause.

Before moving on to capturing logs over the serial console, we noticed that JetPack 7.2 and DeepStream 9.1 had become officially supported for the Jetson Orin series. Instead of continuing the investigation on the older software stack, we decided to upgrade our entire codebase and setup to the latest supported versions.

As part of the migration, we replaced the pyds bindings with the Service Maker Pipeline API (pyservicemaker). Combined with NVIDIA’s deepstream-dev skill set, the migration was surprisingly smooth, requiring only a few manual adjustments.

Our pipeline is now running on DeepStream 9.1, and so far it has been stable for over 38 hours without any unexpected restarts. Based on this, we’re considering the issue resolved on our side for now.

Thanks again for your quick response and support. It’s always appreciated to know that fast and friendly help is available here.