I’m developing a customized AGX Orin platform which uses MAX96712 as a GMSL2 deserializer.
Recently, I found a problem that when the camera is streaming over than 13 hours, the image frames could sometimes arrive later than 33.33ms for a 30 FPS camera. The below log is a real case, you can find “delta: 66.667 ms” there:
> grep -A4 -B4 'seq: 2731202 ' 20240301163737_video0
cap dqbuf: 2 seq: 2731198 bytesused: 5898240 ts: 110376.178919 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq: 2731199 bytesused: 5898240 ts: 110376.212252 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq: 2731200 bytesused: 5898240 ts: 110376.245585 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq: 2731201 bytesused: 5898240 ts: 110376.278918 delta: 33.333 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq: 2731202 bytesused: 5898240 ts: 110376.345585 delta: 66.667 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq: 2731203 bytesused: 5898240 ts: 110376.378918 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq: 2731204 bytesused: 5898240 ts: 110376.412251 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq: 2731205 bytesused: 5898240 ts: 110376.445584 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq: 2731206 bytesused: 5898240 ts: 110376.478917 delta: 33.333 ms fps: 29.88 (ts-monotonic, ts-src-eof)
I cannot reproduce this issue by stress the CPU, RAM, I/O.
So I don’t know where is the root cause? NVIDIA camera RCE firmware?
Need some advice from you, thank you.
Sometimes the delay problem of all the cameras happened at the same time.
All the video start streaming from timestamp 19336, there are no problems between 19336 and 68521 (13hrs) for video4; there are no problems between 19336 and 110000 (25hrs) for video0/video8/vdieo12.
may I know which Jetpack release version you’re working with?
is this related to multi-camera? how many camera streams in your system, had you ever seen the same by only single camera executed?
besides, how you evaluate this delta? is there frame drop issues?
please also check frame-rate reports by adding name=sink_X (to identify the pipeline) for each of your camera streams.
for example, $ gst-launch-1.0 nvarguscamerasrc sensor-id=0 sensor-mode=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v
please see-also Topic 258971, could you please give it a try to add semaphore for capture-ivc.c.
BTW1, did you confirm this cannot reproduced after boosting system clocks?
BTW2, you may double check the sensor driver since you’re testing with v4l2 IOCTL, it’s a very basic utility to communicate with camera drivers.
I’ve given a new test by adding semaphore for capture-ivc.c, boosting system clocks, changing power mode to MAXN, but the problem remains.
And I don’t understand what do you mean for double check the sensor driver. What I want to know first is to distinguish this problem is caused by our hardware design, NVIDIA camera firmware, or v4l-utils itself. Is there a way to find out?
may I know what’s the real defect when you see 66ms?
did you always checking with v4l IOCTL? it’ll output lots of logs after you enable --verbose options. besides, did you have those logs writing to local storage?
let’s setup tegrastats utility to review the usage reports during the issue happened.
BTW,
is it possible for moving to the latest release version, (i.e. JetPack 5.1.3) for verification?
However, the below command, I just use another way to redirect stderr output to log file. It will sometimes cause the delay problem. How to explain this?