ARGUS Capture Stability Issues when stressing CPU

Robotics & Edge Computing Jetson Systems Jetson Orin NX
1

We are getting capture stability issues on a Jetson ORIN NX devkit running JetPack 6.2.1. We don’t have plans to move to a newer version.

The setup for the test is the following:

  1. Enable capture. We tested with a GStreamer pipeline and also with a basic capture application using libnvargus.so. For both we observed the same results:
# Opt1: GStreamer pipeline
gst-launch-1.0 -e \
nvarguscamerasrc silent=false sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12,framerate=30/1' ! \
nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=/application/cam0.mp4 \
nvarguscamerasrc silent=false sensor-id=1 ! \
'video/x-raw(memory:NVMM),width=1920,height=1080,format=NV12,framerate=30/1' ! \
nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=/application/cam1.mp4

# Opt2: Basic Capture app (attached)
./basic-libnvargus 1920 1080 7200
  1. Enable CPU stress command
sudo chrt -f 1 stress-ng --cpu 5 --cpu-load 50
  1. The problem appears after 1-2 minutes:
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/FusaCaptureViCsiHw.cpp, function startCaptureInternal(), line 877)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 547)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 490)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1565)
SCF: Error ResourceAlreadyInUse:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1394)
Acquired frame. 511
SCF: Error Timeout:  (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 644)
SCF: Error Timeout:  (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 430)
SCF: Error ResourceAlreadyInUse:  (propagating from src/common/Utils.cpp, function workerThread(), line 125)
SCF: Error ResourceAlreadyInUse: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 144)
SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 1039)
SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 413)
SCF: Error InvalidState:  (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequest(), line 148)
SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)
[basic-app.tar.gz|attachment](upload://npfNpbZ0szeZJMbU4mIPG2jVjdW.gz) (98.1 KB)

SCF: Error InvalidState: Sending critical error event for Session 0 
 (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
acquireFrame failed: status=6
SCF: Error Timeout:  (propagating from src/components/amr/Snapshot.cpp, function waitForNewerSample(), line 91)
SCF_AutocontrolACSync failed to wait for an earlier frame to complete.

SCF: Error Timeout:  (propagating from src/components/ac_stages/ACSynchronizeStage.cpp, function doHandleRequest(), line 145)
SCF: Error Timeout:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 137)
  1. We tested with the patched libraries found on these posts:
  1. We also ran some tests changing priority to the capture threads:
  • For GStreamer test, we changed nvargus priority to 38 and got better stability:
sudo chrt -f -p 38 $(pgrep nvargus-daemon)
  • The same for the basic:
pid=$(pgrep -x basic-libnvargus)
for task in /proc/"$pid"/task/*; do
    tid=${task##*/}
    name=$(cat "$task/comm")

    case "$name" in
        "SCF Execution"   | \
        "IspHw frameComp"   | \
        "CaptureSchedule"   | \
        "V4L2CaptureSche"   | \
        "cuda-EvtHandlr"    | \
        "PS handleReques"   | \
        "ViCsiHw frameCo"   | \
        "ViCsiHw frameSt"   | \
        "GpuBlitStage"      | \
        "PiecewiseLinear"   | \
        "TempBufferAcqui"   | \
        "CaptureDispatch")
            echo "Setting FIFO 38: TID=$tid [$name]"
            sudo chrt -f -p 38 "$tid"
            ;;
    esac
done

Even if the stability improved with the priority changes, this is not an acceptable solution for this.

Could you please analyze the case and help us improve the stability?

Enrique Ramirez
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com
Website: www.ridgerun.com

basic-app.tar.gz (98.1 KB)

5

The failure appears to originate in VI capture submission, with the later buffer and auto-control timeouts following the capture scheduler failure. Scheduling latency is a strong lead because the stress workload uses SCHED_FIFO and capture priority changes improve stability.

BTW, do you verify by fakesink instead of filesink?

6

Yes, we also tried with fakesink, this simple pipe:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! fakesink

Also, we tested with the attached basic-app.tar.gz from my first message which is a basic capture app. For both I got the failure after a while. We got better stability with the patched libraries, but it’s still failing. So, I’m wondering what on these libraries help to get stability.

I think with this pipe or the basic-app, it will be very easy to reproduce the problem, so if you can try it on your end it would be very helpful as we are looking to improve the capture stability.

8

I am going to try if able to repo the issue.

Thanks