Frame loss occurs when using the nvarguscamera plugin in trig mode

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

hello, JerryChangModerator
I have an agx orin platform, which uses external stm32 to generate pwm waveform to keep the synchronization of the four lenses. However, during the testing of the two lenses, I found the phenomenon of frame loss. The command and test results are as follows:

GST_DEBUG_DUMP_DOT_DIR=/media/Data/log gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540         sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540          ! nvvidconv ! 'video/x-raw(memory:NVMM),width=960,height=1080,format=NV12,framerate=10/1' ! nvv4l2h264enc bitrate=4000000 ! h264parse ! flvmux ! rtmpsink blocksize=24834048  location='rtmp://127.0.0.1/live/mux4'         nvarguscamerasrc sensor-id=2 wbmode=0 ee-mode=0 saturation=0 blocksize=99336192 ! 'video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008,format=(string)NV12, framerate=(fraction)10/1' ! tee name=cam0 !        queue !  nvvidconv  ! "video/x-raw(memory:NVMM), width=2064, height=1504, format=(string)NV12" !  nvvidconv  ! "video/x-raw" !  multifilesink max-file-size=1396915200  next-file=4 blocksize=40960 location=cam0_%04d.nv12 -e cam0. ! queue ! comp.sink_0 cam0. ! queue !   nvvidconv ! 'video/x-raw(memory:NVMM),width=3840,height=2160,format=NV12,framerate=10/1' ! nvv4l2h265enc bitrate=7000000  ! h265parse ! mp4mux ! filesink location=cam-0.mp4         nvarguscamerasrc sensor-id=3 wbmode=0 ee-mode=0 saturation=0 blocksize=99336192 ! 'video/x-raw(memory:NVMM), width=(int)4128, height=(int)3008,format=(string)NV12, framerate=(fraction)10/1' ! tee name=cam1 !         queue !  nvvidconv  ! "video/x-raw(memory:NVMM), width=2064, height=1504, format=(string)NV12" !  nvvidconv  ! "video/x-raw" !  multifilesink max-file-size=1396915200  next-file=4 blocksize=40960 location=cam1_%04d.nv12 -e cam1. ! queue ! comp.sink_1  cam1. ! queue !   nvvidconv ! 'video/x-raw(memory:NVMM),width=3840,height=2160,format=NV12,framerate=10/1'  ! nvv4l2h265enc bitrate=7000000 ! h265parse ! mp4mux ! filesink location=cam-1.mp4

Acquired Frame: 322, time sec 32 msec 6 id:281470807564768
Acquired Frame: 322, time sec 32 msec 18 id:281470782386656
Acquired Frame: 323, time sec 32 msec 105 id:281470807564768
Acquired Frame: 323, time sec 32 msec 118 id:281470782386656
Acquired Frame: 324, time sec 32 msec 206 id:281470807564768
Acquired Frame: 324, time sec 32 msec 217 id:281470782386656
Acquired Frame: 325, time sec 32 msec 305 id:281470807564768
Acquired Frame: 325, time sec 32 msec 317 id:281470782386656
Acquired Frame: 326, time sec 32 msec 404 id:281470782386656
Acquired Frame: 326, time sec 32 msec 417 id:281470807564768
Acquired Frame: 327, time sec 32 msec 504 id:281470807564768
Acquired Frame: 327, time sec 32 msec 516 id:281470782386656
Acquired Frame: 434, time sec 43 msec 244 id:281470782386656
Acquired Frame: 435, time sec 43 msec 361 id:281470807564768
Acquired Frame: 435, time sec 43 msec 382 id:281470782386656
Acquired Frame: 436, time sec 43 msec 420 id:281470807564768
Acquired Frame: 436, time sec 43 msec 426 id:281470782386656
Acquired Frame: 437, time sec 43 msec 473 id:281470807564768
Acquired Frame: 437, time sec 43 msec 486 id:281470782386656

There is a stable trigger signal during the intermediate pauseThere is a stable trigger signal during the intermediate pause.

gstnvarguscamerasrc.cpp

bool StreamConsumer::threadExecute(GstNvArgusCameraSrc *src)
{
      ...
      //if (!src->silent)
      {
        guint64 frame_timestamp = iFrame->getTime() - ground_clk;
        guint64 millisec_timestamp = ((frame_timestamp % (1000000000)))/1000000;
        printf("Acquired Frame: %llu, time sec %llu msec %llu id:%llu\n",
                   static_cast<unsigned long long>(iFrame->getNumber()),
                   static_cast<unsigned long long>(frame_timestamp / (1000000000)),
                   static_cast<unsigned long long>(millisec_timestamp),threadId);
      }
      ...
}

The following is my test of the four-way camera problem MP4, the phenomenon is the same, it is not clear why this situation, CPU/GPU did not run full, and the disk ssd write speed is about 70M, far lower than the maximum write speed supported by ssd.
b7d16a8e25d176e4596df1fde62a2ca3.zip (4.5 MB)

3

hello 1508723374,

it looks stream has stall for a second.
is there any logs from low-level side to indicate frame drop?
for example,
you may setting up a terminal to keep gathering logs by running $ dmesg --follow
and, trying to reproduce this issue again.

4

No. This is the best I can trace it to.

I tried to reproduce the problem, but dmesg didn’t print anything during the time the stream stopped.

5

hello 1508723374,

lets narrow down the issue,
please execute some simplicity gst pipelines for testing.
for example, let’s try put these camera streams as free running, set the camera preview disabled and report frame-rate only.
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

6

The command you gave has been tested several times and there is no stopping the flow.

7

hello 1508723374,

did you meant sensor output streaming continuously?
if yes, it should be something wrong with your app implementation.

8

Yes, I measured the sensor’s signal during the downtime, and there was a data signal coming out. You said that the application of the problem, I are using the system’s own commands and parameters to achieve the function, there will be any problems, can you give some exact tips.

9

hello 1508723374,

let me have double confirmation,
you’re able to execute simple gst pipeline as mentioned in comment #5 without failures?

10

Yeah, you’re right

11

hello 1508723374,

camera functionality should be okay as simple gst pipeline cannot repo the failure,
please narrow down the issue by reduct some actions for your test pipeline, for example, running without filesink or not using nvvidconv to scale the images.

12

I feel that the performance is not enough to cause the delay, is there any performance improvement in this command of mine.

13

hello 1508723374,

you may review and change the power mode with the nvpmodel utility.
furthermore,
you may also try below commands to boost all the VI/CSI/ISP clocks for running camera use-case.
for example,

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
14

I saw the following script in a post earlier, this seems to be the maximum performance of opening orin, and what is the difference between the command you gave?

#!/bin/bash -e
  
if [ $(whoami) != root ]; then
        echo Error: Run this script as a root user
        exit 1
fi

        clkfile=/tmp/defclocks.conf
        pwrfile=/tmp/defpower.conf

        # Xavier
        if [ -e /sys/devices/platform/13e10000.host1x/15340000.vic ]; then
                vicctrl=/sys/devices/platform/13e10000.host1x/15340000.vic
                vicfreqctrl=$vicctrl/devfreq/15340000.vic
        # Orin?
        elif [ -e /sys/devices/platform/13e40000.host1x/15340000.vic ]; then
                vicctrl=/sys/devices/platform/13e40000.host1x/15340000.vic
                vicfreqctrl=$vicctrl/devfreq/15340000.vic
        fi

maxclocks()
{
        if [ ! -e $clkfile ]; then
                jetson_clocks --store $clkfile
                if [ -n "$vicctrl" ]; then
                        echo "$vicfreqctrl/governor:$(cat $vicfreqctrl/governor)" >> $clkfile
                        echo "$vicfreqctrl/max_freq:$(cat $vicfreqctrl/max_freq)" >> $clkfile
                        echo "$vicctrl/power/control:$(cat $vicctrl/power/control)" >> $clkfile
                fi
        fi

        if [ ! -e $pwrfile ]; then
                echo $(nvpmodel -q | tail -n1) > $pwrfile
        fi

        nvpmodel -m 0

        jetson_clocks --fan
        jetson_clocks

        if [ -n "$vicctrl" ]; then
                echo on > $vicctrl/power/control
                echo userspace > $vicfreqctrl/governor
                sleep 1

                maxfreq=$(cat $vicfreqctrl/available_frequencies | rev | cut -f1 -d' ' | rev)

                echo $maxfreq > $vicfreqctrl/max_freq

                echo $maxfreq > $vicfreqctrl/userspace/set_freq

        fi


}
restore()
{
        if [ -e $clkfile ]; then
                jetson_clocks --restore $clkfile > /dev/null 2>&1
        fi



        if [ -e $pwrfile ]; then
                nvpmodel -m $(cat $pwrfile)
        fi

}


action="$1"

case "$action" in
        --restore)
                restore
                ;;
        --max)
                maxclocks
                ;;
        *)
                echo "Unknown option '$action'."
                echo "Usage: $(basename $0) <--max|--restore>"
                exit 1
                ;;
esac
15

it’s all checking sysnode to configure max available clock rate.

16

Sorry for the slow reply. I was on a business trip for work reasons. The situation is that the problem is still unresolved.The configuration of the maximum clock is still not possible, there are some logs I printed below, I do not know if it will help you a little.
not sync.zip (372.3 KB)

17

hello, JerryChangModerator
In the process of debugging I found that the following function does not correspond to the case of frame loss,But I can not continue to debug, where is the implementation of this function? Or where is the implementation of the function that triggers this event?

iEventProvider_ptr->waitForEvents(src->queue.get(), WAIT_FOR_EVENT_TIMEOUT);
18

hello 1508723374,

let’s moving to the new discussion thread since it looks like you’ve submit another new topic for following-up,
for instance, Topic 272499.