[isp][gpu]frame rate drop when gpu overloaded

xingxing · October 17, 2017, 2:54am

The steps we used for our detection algorithm are as below.

Get frame through argus api- > convert to rgb using VIC of low level api → run my algorithm in converterCapturePlaneDqCallback.
The frame rate runs a 30FPS with OV4689+NV ISP.
In normal case, my algorithm cost about 60ms in converterCapturePlaneDqCallback functions. My algorithm is a GPU sensitive type.

So the main loop are, frame(argus) → VIC → converterCapturePlaneDqCallback-> algorithm …frame(argus) → VIC → converterCapturePlaneDqCallback->

Because the algorithm cost 60ms which is longer than the frame rate 30FPS (33.3ms), so the next frame will come just after about 3 ms(mapEGLImage2Float) + 60ms(algorithm).
But now, sometimes it cost about 600+ms to get the next frame. I checked that the algorithm and the mapEGLImage2Float still only need 60 + 3 ms time.
So sounds like the th algorithm (GPU) delayed the next frame in some case.
“cost the difference” indicates the total time the algorithm and mapEGLImage2Float cost.
“the difference” indicates the time cost to wait the next frame.
me.
2. I have test with another case. OV5645 with intergrated ISP inside. It cost about 1000M memory, but when I switch to OV4689 + NV ISP, it cost about 1500M memory.

OV5645 using V4l2 to get frame. OV4689 use the argus api. they run the same algorithm. They both test on the same board just with different sensors.

xingxing · October 17, 2017, 2:58am

Last comment from NVIDIA (Shan Neng Chen - 10/16/2017 7:15 PM):
Hi

Does your pipeline have GPU computing? Can it be reproduced on reference camera board(ov5693)?
Can you provide repro step and app to repo.

Allen_Z · October 17, 2017, 3:09am

Hi, Shan Neng Chen

My algorithm is a deep learning based object detection. The most api it used are cudnn and cuda. It can be reproduced with OV5693 board released from NVIDIA. The only difference is the sensor and it’s driver. BTW the release version is 24.2.1.

2.My algorithm depend on Opencv and other libs. it’s not that easy to directly run on your board.

ShaneCCC · October 17, 2017, 4:37am

@Allen
Could you try skip frames from the deep learning, like maybe compute every third frame only and just bypass the first and second. Or compute every odd frames and bypass even frames.

Allen_Z · October 17, 2017, 6:20am

Hi, ShaneCCC

As you suggested, compute every 3th frame.

RAM 3364/3996MB (lfb 1x4MB) cpu [1%,1%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [2%,2%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [5%,2%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [9%,15%,19%,7%]@1734 GR3D 99%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [13%,20%,9%,13%]@1734 GR3D 86%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [9%,18%,15%,9%]@1734 GR3D 0%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [8%,20%,19%,8%]@1734 GR3D 99%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [9%,18%,22%,13%]@1734 GR3D 70%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [17%,8%,21%,10%]@1734 GR3D 0%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [8%,15%,18%,9%]@1734 GR3D 99%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [11%,13%,23%,6%]@1734 GR3D 45%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [12%,18%,20%,8%]@1734 GR3D 0%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [17%,12%,16%,6%]@1734 GR3D 99%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [16%,21%,8%,12%]@1734 GR3D 26%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [11%,23%,15%,11%]@1734 GR3D 0%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [6%,25%,14%,10%]@1734 GR3D 99%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [20%,15%,12%,14%]@1734 GR3D 17%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [11%,18%,4%,18%]@1734 GR3D 0%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [10%,19%,10%,22%]@1734 GR3D 99%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [13%,12%,6%,18%]@1734 GR3D 7%@998 EDP limit 0
RAM 3363/3996MB (lfb 1x4MB) cpu [12%,17%,8%,22%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [12%,18%,17%,14%]@1734 GR3D 99%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [18%,15%,12%,15%]@1734 GR3D 3%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [15%,13%,8%,13%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [3%,1%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [1%,1%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [2%,0%,2%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [4%,4%,3%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [3%,1%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [1%,4%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [4%,5%,2%,0%]@1734 GR3D 9%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [5%,3%,3%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [2%,1%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [2%,3%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [4%,0%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [4%,4%,3%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [3%,1%,2%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [2%,3%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [2%,2%,2%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [1%,1%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [3%,1%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3365/3996MB (lfb 1x4MB) cpu [18%,14%,11%,13%]@1734 GR3D 99%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [11%,17%,12%,17%]@1734 GR3D 16%@998 EDP limit 0
RAM 3364/3996MB (lfb 1x4MB) cpu [3%,6%,3%,6%]@1734 GR3D 0%@998 EDP limit 0

ShaneCCC · October 17, 2017, 6:30am

@Allen
Still got frame drop after cuda skip frames? What information tell from the log?

Allen_Z · October 17, 2017, 6:34am

Yes, still blocked the frame stream as before.

RAM 3252/3996MB (lfb 2x2MB) cpu [1%,1%,2%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [7%,2%,4%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [5%,1%,6%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,2%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,3%,5%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,0%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [7%,1%,4%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [0%,0%,2%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,1%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,1%,4%,4%]@1734 GR3D 98%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,4%,0%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,0%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [6%,1%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [8%,4%,2%,8%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,0%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,4%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,2%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [7%,1%,5%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,1%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,0%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [7%,5%,3%,2%]@1734 GR3D 99%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,3%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,2%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,2%,2%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [6%,1%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [6%,3%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,0%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [3%,2%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [9%,4%,4%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,0%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,2%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [5%,1%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,0%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [3%,1%,2%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [3%,1%,2%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [20%,17%,11%,14%]@1734 GR3D 68%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [18%,5%,9%,7%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,0%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,1%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [7%,4%,3%,0%]@1734 GR3D 6%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [10%,6%,2%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,2%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,2%,1%,2%]@1734 GR3D 7%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,3%,2%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,1%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,1%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [11%,14%,8%,9%]@1734 GR3D 99%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [21%,12%,17%,12%]@1734 GR3D 59%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [16%,14%,21%,15%]@1734 GR3D 99%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [18%,19%,9%,14%]@1734 GR3D 99%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [17%,14%,7%,22%]@1734 GR3D 92%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [0%,4%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,2%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,1%,2%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,3%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,1%,2%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [5%,3%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [24%,19%,15%,10%]@1734 GR3D 53%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [12%,21%,25%,11%]@1734 GR3D 99%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [10%,10%,13%,21%]@1734 GR3D 98%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [11%,18%,13%,21%]@1734 GR3D 96%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [7%,13%,5%,11%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,0%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,2%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,1%,1%,5%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [5%,4%,3%,6%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,3%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [7%,2%,4%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,3%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,2%,3%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,2%,1%,5%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,1%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,2%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [5%,5%,1%,5%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [1%,2%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [4%,2%,0%,5%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,1%,3%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [6%,7%,6%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [0%,2%,3%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [1%,0%,4%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [5%,6%,3%,5%]@1734 GR3D 99%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,1%,2%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,2%,2%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [4%,2%,3%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [8%,4%,0%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,3%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,1%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [1%,3%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [3%,2%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3252/3996MB (lfb 2x2MB) cpu [2%,1%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [16%,16%,11%,14%]@1734 GR3D 93%@998 EDP limit 0
RAM 3251/3996MB (lfb 2x2MB) cpu [15%,13%,21%,27%]@1734 GR3D 99%@998 EDP limit 0

[16323.733864] gain128 = 104
[16323.734930] ov4689 30-0036: ov4689_set_coarse_time: val: 7620
[16324.415552] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16324.422476] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16324.430085] ov4689 30-0036: could not find ctrl 9a2002
[16324.435340] gain128 = 374
[16324.436409] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16325.097130] gain128 = 354
[16325.278721] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16325.285921] ov4689 30-0036: could not find ctrl 9a2002
[16325.291145] gain128 = 103
[16325.292072] ov4689 30-0036: ov4689_set_coarse_time: val: 6350
[16325.324209] gain128 = 102
[16326.006042] gain128 = 101
[16326.687603] gain128 = 101
[16327.369280] gain128 = 110
[16327.370480] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16328.050930] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16328.060311] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16328.067496] ov4689 30-0036: could not find ctrl 9a2002
[16328.072892] gain128 = 332
[16328.073856] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16328.914087] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16328.922587] ov4689 30-0036: could not find ctrl 9a2002
[16328.927802] gain128 = 110
[16328.928727] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16334.412800] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16334.421007] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16334.428683] ov4689 30-0036: could not find ctrl 9a2002
[16334.433881] gain128 = 332
[16334.435174] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16336.457196] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16336.465528] ov4689 30-0036: could not find ctrl 9a2002
[16336.470714] gain128 = 110
[16336.471637] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16339.910970] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16339.917805] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16339.925309] ov4689 30-0036: could not find ctrl 9a2002
[16339.930648] gain128 = 332
[16339.931513] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16340.819604] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16340.826660] ov4689 30-0036: could not find ctrl 9a2002
[16340.831822] gain128 = 110
[16340.832737] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16345.636704] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16345.643994] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16345.651714] ov4689 30-0036: could not find ctrl 9a2002
[16345.657088] gain128 = 332
[16345.658377] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16350.769786] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16350.777399] ov4689 30-0036: could not find ctrl 9a2002
[16350.782567] gain128 = 110
[16350.783422] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16354.725922] DHCP - REQUEST [RX]
[16356.268381] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16356.275346] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16356.282782] ov4689 30-0036: could not find ctrl 9a2002
[16356.288049] gain128 = 332
[16356.289341] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16361.401515] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16361.408483] ov4689 30-0036: could not find ctrl 9a2002
[16361.413660] gain128 = 110
[16361.414791] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16364.855351] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16364.863753] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16364.871328] ov4689 30-0036: could not find ctrl 9a2002
[16364.876621] gain128 = 332
[16364.877628] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16365.718522] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16365.726640] ov4689 30-0036: could not find ctrl 9a2002
[16365.731957] gain128 = 110
[16365.732867] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16368.490742] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16368.498427] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16368.505949] ov4689 30-0036: could not find ctrl 9a2002
[16368.511098] gain128 = 332
[16368.512358] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16369.353978] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16369.362350] ov4689 30-0036: could not find ctrl 9a2002
[16369.367807] gain128 = 110
[16369.368879] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16372.807810] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16372.815742] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16372.823658] ov4689 30-0036: could not find ctrl 9a2002
[16372.828870] gain128 = 332
[16372.829720] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16373.671109] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16373.677984] ov4689 30-0036: could not find ctrl 9a2002
[16373.683246] gain128 = 110
[16373.684122] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16376.443023] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16376.450256] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16376.457875] ov4689 30-0036: could not find ctrl 9a2002
[16376.463032] gain128 = 332
[16376.464209] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16377.351739] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16377.360922] ov4689 30-0036: could not find ctrl 9a2002
[16377.366108] gain128 = 110
[16377.366960] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16380.123951] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16380.131333] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16380.138791] ov4689 30-0036: could not find ctrl 9a2002
[16380.144021] gain128 = 332
[16380.145089] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16381.032553] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16381.040915] ov4689 30-0036: could not find ctrl 9a2002
[16381.046470] gain128 = 110
[16381.047336] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16383.804805] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16383.811833] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16383.818967] ov4689 30-0036: could not find ctrl 9a2002
[16383.824397] gain128 = 332
[16383.825409] ov4689 30-0036: ov4689_set_coarse_time: val: 1905
[16384.667998] ov4689 30-0036: ov4689_set_frame_length: val: 31754
[16384.675833] ov4689 30-0036: could not find ctrl 9a2002
[16384.681241] gain128 = 110
[16384.682127] ov4689 30-0036: ov4689_set_coarse_time: val: 5715
[16389.485155] ov4689 30-0036: ov4689_set_frame_length: val: 2116
[16389.493517] ov4689 30-0036: ov4689_set_coarse_time: val: 2110
[16389.501085] ov4689 30-0036: could not find ctrl 9a2002
[16389.506298] gain128 = 332
[16389.507435] ov4689 30-0036: ov4689_set_coarse_time: val: 1905

ShaneCCC · October 17, 2017, 6:59am

@Allen
How do you check the frame drop from these logs?

Allen_Z · October 17, 2017, 7:17am

In normal state, When I run with OV5645 with the same algorithm, the GPU is always in full load state.

Allen_Z · October 17, 2017, 7:18am

Also I can see the video from the screen through hdmi.

ShaneCCC · October 17, 2017, 7:46am

@Allen

Does any improve for the skip frames? How about increase the number of the skip frame.
Does the skip frames still show to display?
Could you try isolation the cuda code to process a static frame with the camera pipeline to check if the still have frame drop from the camera?

Allen_Z · October 18, 2017, 4:19am

Hi, ShanbeCCC

Skip with different frames.

skip every 3 frames.

RAM 3257/3996MB (lfb 3x4MB) cpu [10%,1%,4%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [4%,12%,3%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [2%,0%,2%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [6%,1%,4%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [4%,4%,2%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [3%,0%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [3%,4%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [2%,2%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [9%,2%,2%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [2%,0%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [3%,1%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [8%,4%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [6%,5%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [3%,4%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3257/3996MB (lfb 3x4MB) cpu [5%,1%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [5%,1%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [8%,3%,1%,1%]@1734 GR3D 99%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [1%,1%,2%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [1%,1%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [6%,4%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [2%,2%,0%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [5%,4%,0%,0%]@1734 GR3D 99%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [5%,2%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3258/3996MB (lfb 3x4MB) cpu [4%,2%,2%,3%]@1734 GR3D 0%@998 EDP limit 0

skip every 6 frames.

pu [6%,3%,2%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [2%,0%,0%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [4%,1%,2%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [1%,1%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [8%,3%,2%,1%]@1734 GR3D 53%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,1%,1%,0%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [2%,11%,0%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [6%,1%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [5%,2%,5%,8%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,0%,0%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [2%,1%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,1%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [5%,1%,2%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [4%,1%,1%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [1%,0%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [7%,2%,2%,5%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,1%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [4%,1%,0%,3%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,1%,1%,2%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,0%,1%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [4%,4%,3%,4%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,0%,0%,1%]@1734 GR3D 0%@998 EDP limit 0
RAM 3266/3996MB (lfb 2x4MB) cpu [3%,0%,1%,2%]@1734 GR3D 0%@998 EDP limit 0

Skip all the frames, mean no detection will run via stream loop. about 2% percent gpu will be load because the ISP.

RAM 2970/3996MB (lfb 58x4MB) cpu [8%,9%,9%,18%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [12%,7%,7%,16%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [13%,15%,11%,23%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [12%,8%,9%,19%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [8%,10%,11%,20%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [10%,8%,5%,15%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [11%,13%,10%,12%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [10%,10%,9%,16%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [15%,5%,10%,19%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [9%,12%,12%,12%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [11%,9%,12%,13%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [8%,7%,14%,14%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [7%,9%,11%,15%]@1734 GR3D 0%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [9%,9%,10%,17%]@1734 GR3D 2%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [9%,9%,6%,15%]@1734 GR3D 0%@998 EDP limit 0
RAM 2971/3996MB (lfb 58x4MB) cpu [10%,9%,10%,15%]@1734 GR3D 3%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [12%,7%,9%,17%]@1734 GR3D 3%@998 EDP limit 0
RAM 2971/3996MB (lfb 58x4MB) cpu [8%,10%,11%,18%]@1734 GR3D 4%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [15%,10%,10%,13%]@1734 GR3D 5%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [9%,8%,5%,18%]@1734 GR3D 5%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [12%,5%,12%,15%]@1734 GR3D 5%@998 EDP limit 0
RAM 2971/3996MB (lfb 58x4MB) cpu [9%,6%,12%,16%]@1734 GR3D 4%@998 EDP limit 0
RAM 2971/3996MB (lfb 58x4MB) cpu [9%,10%,7%,18%]@1734 GR3D 4%@998 EDP limit 0
RAM 2971/3996MB (lfb 58x4MB) cpu [10%,10%,16%,8%]@1734 GR3D 5%@998 EDP limit 0
RAM 2971/3996MB (lfb 58x4MB) cpu [9%,11%,14%,8%]@1734 GR3D 2%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [11%,13%,11%,7%]@1734 GR3D 2%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [13%,15%,10%,7%]@1734 GR3D 2%@998 EDP limit 0
RAM 2970/3996MB (lfb 58x4MB) cpu [11%,15%,11%,9%]@1734 GR3D 1%@998 EDP limit 0
RAM 2971/3996MB (lfb 58x4MB) cpu [7%,17%,12%,8%]@1734 GR3D 0%@998 EDP limit 0

The log of no algorithm running, just stream.

CONSUMER: Done.
PRODUCER: Done -- exiting.
----------- Element =  -----------
Total Profiling time = 12.4847
Average FPS = 22.0269
Total units processed = 276
Average latency(usec) = 1060
Minimum latency(usec) = 870
Maximum latency(usec) = 2551
-------------------------------------
PowerServiceCore:handleRequests: timePassed = 2011

(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)

Yes, the display is always there.
frame drop is still there.

Allen_Z · October 18, 2017, 4:22am

The log of algorithm running in every frame.
in normal state with ov5645 the same algorithm will has about 19 frame rate.

CONSUMER: Done.
PRODUCER: Done -- exiting.
----------- Element =  -----------
Total Profiling time = 41.7613
Average FPS = 2.4664
Total units processed = 104
Average latency(usec) = 105708
Minimum latency(usec) = 913
Maximum latency(usec) = 1068228
-------------------------------------
PowerServiceCore:handleRequests: timePassed = 2022

(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)
(NvOdmDevice) Error NotInitialized: V4L2Device not powered on (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function setControlVal(), line 378)

ShaneCCC · October 18, 2017, 5:48am

@Allen

Do you still see the frame drop for skip 6 frames? The CPU, GPU usage can’t tell if any frame drop or not. Please have detail information for your experiment.
What’s the frame rate for this sensor?
Please have more explain for your latest comment "The log of algorithm running in every frame.
in normal state with ov5645 the same algorithm will has about 19 frame rate.
"

Allen_Z · October 19, 2017, 4:34am

@ShaneCCC

I found the root cause.

We placed a dbus communication api in capture_callback of the vic_convert to notice the current arriving of the new stream.
It used to be a non-blocking api cost less that 1ms in the pre version.
But now it has been updated to a synchronization api,
so when the dbus system is heavy especially there are detection result sending to the dbus core frequently, the blocking time will be increased to a long time.
Just looks like when the algorithm is running the the delayed time increased.
When we fallback to the pre-version, the problem sovled. So this bug is root from the blocking dbus api we used. It was ignored in my verification.

@ShaneCCC @xingxing

Thanks very much for all your support.