Jetpack6.2_Cameras can't receive streaming after system run full load a whole time

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Hi Nvidia,

My devices :

NVIDIA® Jetson AGX Orin™ Developer Kit *1

LI-JAG-ADP-GMSL2-8CH V2.0 *1

OWL cameras *4

HAWK cameras *4

reference : LI-AGO-ADP-HAWK-OWL-Q - Leopard Imaging Inc.

I use “argus_camera” to open all cameras and use “stress-ng“ to setting CPU/GPU loading (like CPU loading 100% , GPU loading 50%) , after system run a whole time , all cameras will stop streaming (sometimes not all cameras stop streaming), how should I fix this issue?

jetosn_orin_agx_jp62_dmesg.txt (112.9 KB)

3

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing with your NVIDIA Jetson AGX Orin Developer Kit, where all cameras stop streaming after a period of time, is likely related to power and performance management on the device. Here’s a comprehensive analysis and potential solutions:

Power and Performance Management on Jetson Devices

The NVIDIA Jetson AGX Orin series devices have power, thermal, and electrical management features visible to software. These features are crucial in managing the device’s performance and preventing overheating, which can cause camera streaming issues.

Possible Causes

  1. Thermal Throttling: The Jetson AGX Orin device may be experiencing thermal throttling, where the system reduces performance to prevent overheating. This can cause camera streaming to stop.
  2. Power Management: The device’s power management system may be limiting the power available to the cameras, causing them to stop streaming.
  3. Resource Contention: With multiple cameras and high CPU/GPU loading, resource contention may be occurring, leading to camera streaming issues.

Potential Solutions

  1. Monitor System Temperature: Use tools like tmon or nvpmodel to monitor the system temperature and adjust the cooling system accordingly.
  2. Adjust Power Settings: Use the nvpmodel tool to adjust the power settings and allocate more power to the cameras.
  3. Optimize Resource Allocation: Use tools like stressng to optimize resource allocation and reduce contention between cameras and other system components.
  4. Update Device Firmware: Ensure that the device firmware is up-to-date, as newer versions may include performance and power management improvements.
  5. Camera Configuration: Verify that the camera configuration is optimal for the system’s power and performance capabilities.

Comparison with Other Jetson Devices

The Jetson AGX Orin series devices have similar power and performance management features to other Jetson devices, such as the Jetson Xavier NX and Jetson AGX Xavier series. However, the specific power and performance management features may vary between devices.

Additional Recommendations

  1. Review System Logs: Review the system logs (e.g., dmesg) to identify any error messages or warnings related to power, thermal, or camera issues.
  2. Test with Different Camera Configurations: Test the system with different camera configurations to isolate the issue and identify potential workarounds.
  3. Consult NVIDIA Documentation: Consult the NVIDIA Jetson AGX Orin documentation and developer guides for more information on power and performance management, as well as camera development.

By following these steps and considering the potential causes and solutions, you should be able to identify and fix the issue with your NVIDIA Jetson AGX Orin Developer Kit.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

hello james.hsu2

may I know what’s the actual use-case to simulate CPU/GPU fully occupied?

5

Hi Jerry,

Among our clients, if 12 cameras are used in a busy system (like calculate data), there is a possibility that some cameras may experience streaming interruptions.

6

hello james.hsu2,

please refer to Supported Modes and Power Efficiency to configure system running at MaxN (i.e. performance mode)
besides..
according to the attached logs, there’s no error messages reported. it looks like userspace to close the capture app, please try again with serial console to gather the logs while issue happened.

7

How should I debug this? I tried adding logs in nv_hawk_owl.c/capture-vi.c/csi5_fops.c to debug I2C and CSI, but all I got was a call to ar0234_stop_streaming(). Do you have any suggestions?

8

hello james.hsu2,

please refer to Camera Architecture Stack, libargus and v4l2 IOCTL they’re went through different pipelines.
since you’re running with argus_camera app, please collect nvargus-daemon logs with.. $ sudo journalctl -b -u nvargus-daemon

9

Hi Jerry,

I received the following error log, along with a log.txt file.

jetosn_agx_orin_jp62_argus_camera_log.txt (9.5 KB)

Dec 24 12:29:35 NAC1k-NovaOrin argus_camera[913932]: Module_id 30 Severity 2 : (fusa) Error: InvalidState Status syncpoint signaled but status value not updated in:/capture/src/fusaViHandler.cpp 861
Dec 24 12:29:35 NAC1k-NovaOrin argus_camera[913932]: Module_id 30 Severity 2 : (fusa) Error: InvalidState propagating from:/capture/src/fusaViHandler.cpp

13

Hi @JerryChang,

Any updates on the fusa error message? Or should I collect other logs?