Hi Team,
I am working with four IMX676 sensors connected via the CSI interface on an Orin NX.
- In V4L2, the cameras passed our continuous stream on/off test successfully for 2 days (I stopped the test manually).
- In Argus, the cameras also passed continuous long streaming without any issues.
- However, when running the continuous stream on/off test in Argus, I encounter stability issues.
For Argus testing, I used the following GStreamer command using script which I’m attaching below for your reference in the name ‘continuous_stream_on_off.txt’.
argus.log (22.9 KB)
gstreamer.log (14.8 KB)
continuous_stream_on_off.txt (436 Bytes)
I have also attached both the Argus log and the GStreamer log for your reference.
Could you please help analyze why Argus fails in the on/off stability test while V4L2 and long continuous streaming both pass? Any suggestions for further debugging or tuning would be very helpful.
Thanks in advance,
Adam
1 Like
*** Please note that this reply is generated by LLM automatically ***
Based on the provided information, I’ll try to help analyze the issue and provide suggestions for further debugging or tuning.
Summary of the issue:
You’re experiencing stability issues with Argus when running a continuous stream on/off test with four IMX676 sensors connected via the CSI interface on an Orin NX. The test passes successfully in V4L2 and with long continuous streaming in Argus, but fails with the on/off test in Argus.
Possible causes:
- ISP configuration: The Camera Core release package includes model ISP configuration files for reference sensors. However, it’s possible that the ISP configuration for the IMX676 sensors is not optimal or is causing issues with the on/off test. You may want to review the ISP configuration files and adjust them if necessary.
- Virtual channel configuration: The Jetson AGX Orin series supports the MIPI CSI virtual channel feature, which allows for multiplexed sensor streams to share the same CSI port/brick and CSI stream. However, it’s possible that the virtual channel configuration is not correct or is causing issues with the on/off test. You may want to review the virtual channel configuration and adjust it if necessary.
- GStreamer pipeline: The GStreamer pipeline used for the on/off test may be causing issues. You may want to review the pipeline and adjust it if necessary. For example, you could try using a different pipeline or adjusting the buffer sizes.
- Sensor driver issues: The sensor driver may be causing issues with the on/off test. You may want to review the sensor driver code and adjust it if necessary.
Suggestions for further debugging or tuning:
- Review the Argus log and GStreamer log: Carefully review the Argus log and GStreamer log to see if there are any error messages or clues that can help identify the issue.
- Check the ISP configuration: Review the ISP configuration files and adjust them if necessary.
- Verify the virtual channel configuration: Review the virtual channel configuration and adjust it if necessary.
- Adjust the GStreamer pipeline: Try adjusting the GStreamer pipeline to see if it resolves the issue.
- Test with a different sensor: Try testing with a different sensor to see if the issue is specific to the IMX676 sensors.
- Test with a different CSI port: Try testing with a different CSI port to see if the issue is specific to the current CSI port.
- Monitor system resources: Monitor system resources such as CPU usage, memory usage, and temperature to see if there are any issues that could be causing the stability problems.
Comparison with V4L2:
V4L2 is a lower-level API than Argus, and it’s possible that the V4L2 test is not exercising the same functionality as the Argus test. However, the fact that the V4L2 test passes successfully suggests that the issue is likely specific to Argus or the GStreamer pipeline.
I hope these suggestions help you identify and resolve the issue. If you have any further questions or need more specific guidance, please let me know.
*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***
Hi,
Everything appears to be configured correctly on my side, and the issue only occurs during long-run testing. I have attached the logs for your review. Could you please go through them and suggest how I can proceed with further debugging? Looking forward to your guidance.
Check below patch.
[Argus] Long run stability issue.
https://forums.developer.nvidia.com/t/318999/5
Hi @ShaneCCC,
I followed the recommendation to replace the two library files and verified by running the test again. However, the issue still occurs when using 4 cameras after around 12 hours of stream on and off operation. I am seeing the same behaviour as before.
From the Argus log during the failure, I observed the following same error as attached before:
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 969)
I am using the same GStreamer command as mentioned in my previous message. I’m using 4 cameras in my setup.
I also checked by using the below commands for Infinite Timeout Support but it didn’t work as well.
sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon
Could you kindly help us further in resolving this issue? Looking forward to your guidance.
Could you modify the CID function in sensor driver like xxx_set_exposure to dummy function to verify.
Hi @ShaneCCC,
Thank you for the suggestion.
I modified (all CID functions like xxx_set_exposure) in the sensor driver to a dummy function as advised and reran the long-duration stream on/off test. Unfortunately, the same issue still occurs. I continue to see the same timeout errors as before, with no improvement in stability.
Could you please advise what I should check next to resolve this issue?
Please also apply below.
[gpu/host1x-fence] Significant kernel memory leak when using Argus camera
https://forums.developer.nvidia.com/t/325399/4/
[Argus][multi-cam] getModuleString() always returns the values of 1st camera
https://forums.developer.nvidia.com/t/343974/6/
Hi @ShaneCCC,
Thank you for the suggestions.
I have applied both of the recommended patches:
However, I still encounter the same issue after few hours of continuous stream on & off testing. When the issue occurs, I can recover the streaming only by restarting the Argus service using:
sudo service nvargus-daemon restart
This has been the only way to restore functionality from the beginning.
Could you please suggest any further debugging steps to help identify and resolve the root cause without restarting the Argus daemon?
Get the daemon log.
journalctl -u nvargus-daemon -f
Hi @ShaneCCC,
Kindly review the attached logs and let me know your observations or the next steps for debugging.
nvargus-daemon.log (5.6 MB)
What’s the eCAM_argus_camera?
The eCAM_argus_camera is our custom application.
I used it initially to verify that all the cameras were working correctly before starting the test script.
The actual test script uses a GStreamer pipeline for streaming the cameras during the long-run test. GStreamer command shared in my initial post in this thread in the name ‘continuous_stream_on_off.txt’.
Please try use num-buffers=150 to terminate instead of kill command.
#RUNNING THE BELOW CODE SNIPPET IN CONTINUOUS LOOP
for cam in $(seq 0 $((camera_count - 1))); do # STREAM ON EACH CAMERA WITH 1 SECOND DELAY BETWEEN EACH CAMERA
gst-launch-1.0 nvarguscamerasrc sensor-id=${cam} num-buffers=150 ! "video/x-raw(memory:NVMM), width=3520,height=3520, framerate=15/1, format=(string)RG12" ! queue ! nv3dsink -e -v 2>&1 &
sleep 1
done
sleep 20 # STREAM FOR 5 SECONDS
#killall -15 gst-launch-1.0 2>&1
#sleep 3
Hi @ShaneCCC,
Thank you for the suggestion.
I have tested using num-buffers=150 to allow the stream to terminate naturally instead of using the kill command.
However, I still encountered the same issue after few hours of continuous stream on & off testing.
The stream could only be recovered by restarting the Argus daemon.
I have attached both the dmesg and Argus daemon logs below for your reference.
Kindly review them and let me know your observations or any further steps I can take for debugging.
argus.log (6.6 MB)
dmesg.log (438.2 KB)
Could you check the soc temperature.
Maybe check if able keep it cooler to confirm below message.
Oct 09 17:48:01 nvidia nvargus-daemon[1007]: Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame: 167
Oct 09 17:48:01 nvidia nvargus-daemon[1007]: Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
Oct 09 17:48:01 nvidia nvargus-daemon[1007]: Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
Oct 09 17:48:01 nvidia nvargus-daemon[1007]: Module_id 30 Severity 2 : FuSaCaptureViErrorDecode VI Error in frame: 199
Oct 09 17:48:01 nvidia nvargus-daemon[1007]: Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED 0x000009
Oct 09 17:48:01 nvidia nvargus-daemon[1007]: Module_id 30 Severity 2 : ATOMP_FRAME_TRUNCATED: Frame not finished
Hi @ShaneCCC,
Thank you for the suggestion.
Previously, the fan RPM was set to less than 50%.
I have now used the following command to run the fan at 100% PWM and also added an external fan for additional cooling:
sudo jetson_clocks --fan
Earlier, the SoC temperature was around 59°C, and after applying full fan speed with external cooling, it has reduced to around 44°C.
I have attached the temperature images for your reference, which were taken after enabling 100% fan PWM and using the external fan.
I have also captured the tegrastats log during the test for your reference:
10-13-2025 22:06:59 RAM 7743/15656MB (lfb 9x4MB) SWAP 0/7828MB (cached 0MB) CPU [22%@1984,16%@1984,16%@1984,16%@1984,13%@1984,14%@1984,24%@1984,17%@1984] EMC_FREQ 22%@3199 GR3D_FREQ 23%@[917] NVENC off NVDEC off NVJPG off NVJPG1 off VIC 99%@729 OFA off NVDLA0 off NVDLA1 off PVA0_FREQ off APE 174 cv0@42.062C cpu@43.75C soc2@42.562C soc0@42.718C cv1@42.187C gpu@42.968C tj@44.562C soc1@44.562C cv2@42.406C VDD_IN 13600mW/8453mW VDD_CPU_GPU_CV 3389mW/2582mW VDD_SOC 6339mW/3272mW
10-13-2025 22:07:00 RAM 7746/15656MB (lfb 9x4MB) SWAP 0/7828MB (cached 0MB) CPU [19%@1984,19%@1984,17%@1984,13%@1984,17%@1984,12%@1984,19%@1984,12%@1984] EMC_FREQ 23%@3199 GR3D_FREQ 37%@[917] NVENC off NVDEC off NVJPG off NVJPG1 off VIC 99%@729 OFA off NVDLA0 off NVDLA1 off PVA0_FREQ off APE 174 cv0@42.125C cpu@43.843C soc2@42.75C soc0@42.843C cv1@42.406C gpu@42.968C tj@44.687C soc1@44.687C cv2@42.593C VDD_IN 13560mW/8753mW VDD_CPU_GPU_CV 3309mW/2625mW VDD_SOC 6379mW/3455mW
However, I still observe the same Argus log messages.
Kindly review and suggest the next steps for debugging.
Could you confirm this error log shows by launch single camera?
