Receive image frame later than expected

Hi Guys,

I’m developing a customized AGX Orin platform which uses MAX96712 as a GMSL2 deserializer.

Recently, I found a problem that when the camera is streaming over than 13 hours, the image frames could sometimes arrive later than 33.33ms for a 30 FPS camera. The below log is a real case, you can find “delta: 66.667 ms” there:

> grep -A4 -B4 'seq: 2731202 ' 20240301163737_video0
cap dqbuf: 2 seq: 2731198 bytesused: 5898240 ts: 110376.178919 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq: 2731199 bytesused: 5898240 ts: 110376.212252 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq: 2731200 bytesused: 5898240 ts: 110376.245585 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq: 2731201 bytesused: 5898240 ts: 110376.278918 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq: 2731202 bytesused: 5898240 ts: 110376.345585 delta: 66.667 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq: 2731203 bytesused: 5898240 ts: 110376.378918 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq: 2731204 bytesused: 5898240 ts: 110376.412251 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq: 2731205 bytesused: 5898240 ts: 110376.445584 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq: 2731206 bytesused: 5898240 ts: 110376.478917 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)

I cannot reproduce this issue by stress the CPU, RAM, I/O.
So I don’t know where is the root cause? NVIDIA camera RCE firmware?
Need some advice from you, thank you.

The below logs saved only the error frames, so the sequence is not coherent.
video0, video4, video8, and video12 are connected to different MAX96712 and different VI streams.
20240301163737_video0_result.txt (16.6 KB)
20240301163737_video4_result.txt (24.6 KB)
20240301163737_video8_result.txt (16.2 KB)
20240301163737_video12_result.txt (17.8 KB)

After observing the logs, here are what I found:

  • Sometimes the delay problem of all the cameras happened at the same time.
  • All the video start streaming from timestamp 19336, there are no problems between 19336 and 68521 (13hrs) for video4; there are no problems between 19336 and 110000 (25hrs) for video0/video8/vdieo12.

hello ting.chang,

may I know which Jetpack release version you’re working with?
is this related to multi-camera? how many camera streams in your system, had you ever seen the same by only single camera executed?

besides, how you evaluate this delta? is there frame drop issues?
please also check frame-rate reports by adding name=sink_X (to identify the pipeline) for each of your camera streams.
for example,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

Hi @JerryChang

Actually, the delta is calculated by v4l2-ctl itself, we can get the delta values by adding --verbose option to v4l2-ctl.

v4l2-ctl --stream-mmap -d /dev/video0 --verbose

I have checked the logs, the sequences are all coherent, and no drop error is reported by v4l2.

I will test only one camera to see if the issue remains.

After testing the single camera, the problem remains.

The new findings:

  • When the system is not rebooted. We close stream and start stream again for a new test. The problem happens within 30 minutes.
  • However, when the system has been rebooted, the problem happens after 12 or 24 hours.

hello ting.chang,

may I also know which Jetpack release version you’re working with?

Hi @JerryChang

L4T: R35.4.1
JetPack 5.1.2
AGX Orin 64G

hello ting.chang,

please see-also Topic 258971, could you please give it a try to add semaphore for capture-ivc.c.

BTW1, did you confirm this cannot reproduced after boosting system clocks?
BTW2, you may double check the sensor driver since you’re testing with v4l2 IOCTL, it’s a very basic utility to communicate with camera drivers.

Hi @JerryChang

I’ve given a new test by adding semaphore for capture-ivc.c, boosting system clocks, changing power mode to MAXN, but the problem remains.

And I don’t understand what do you mean for double check the sensor driver. What I want to know first is to distinguish this problem is caused by our hardware design, NVIDIA camera firmware, or v4l-utils itself. Is there a way to find out?

hello ting.chang,

it’s not related to software stack as you’ve check there’s no frame drops, and the capture sequence it’s in orders.

Hi @JerryChang

“Sometimes the delay problem of all the cameras happened at the same time.”
I think it may relate to software stack. But I don’t know how to prove it.

hello ting.chang,

may I know what’s the real defect when you see 66ms?

did you always checking with v4l IOCTL? it’ll output lots of logs after you enable --verbose options. besides, did you have those logs writing to local storage?
let’s setup tegrastats utility to review the usage reports during the issue happened.

BTW,
is it possible for moving to the latest release version, (i.e. JetPack 5.1.3) for verification?

Hi @JerryChang

This command tested for over 3days without any problem.

v4l2-ctl --set-fmt-video=width=1920,height=1536 --stream-mmap --stream-count=0 -d /dev/video0 --verbose 2> >(tee log_video0)

However, the below command, I just use another way to redirect stderr output to log file. It will sometimes cause the delay problem. How to explain this?

v4l2-ctl --set-fmt-video=width=1920,height=1536 --stream-mmap --stream-count=0 -d /dev/video0 --verbose 2> log_video0

hello ting.chang,

there do have some advantages of using tee. you may see-also below for reference.
shell - What is the purpose of 'tee'? - Super User

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.