To understand this problem better, I add another log in the encoderCapturePlaneDqCallback function, which should be called when a frame is finished encoding and waits inside a capture plane. We got the following result (I put some orange comments to help us understand):
CONSUMER: Acquired Frame. 1828717843//1st frame acquired from the camera
CONSUMER: Acquired Frame. 1828717844//2nd frame acquired from the camera
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 22.//This is probably some meta data bscause it is only 22 bytes.
CONSUMER: Acquired Frame. 1828717850//3rd frame acquired from the camera
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 26762.//1st frame finishes encoding, which happens after 3rd frame from camera is ready for encoding.
CONSUMER: Acquired Frame. 1828717853//4th frame acquired from the camera
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 24245.//2nd frame finishes encoding, which happens after 4th frame from camera is ready for encoding.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 23706.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 24028.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 7422.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 13760.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 26142.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 35054.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30708.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 32713.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30744.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 33689.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 34431.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30678.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 31665.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 29837.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30142.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 31853.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 25620.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 26251.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 23094.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 23194.
CONSUMER: Acquired Frame. 1828717868//the last frame acquired from the camera
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 24220.//the last frame minus 2 finishes encoding
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 25075.//the last frame minus 1 finishes encoding
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 19293.//the last frame finishes encoding
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 0.
CONSUMER: Got EOS, exiting…
CONSUMER: Done.
From this result, we verify the argument that each frame will finish encoding after 2 frame periods + some time. Since I’m capturing 30fps video with 640x480 resolution, the time of 2 frame periods is 66.7ms. The same question: So can we really force every frame to be decoded straight away without waiting for the next frame?
Current high-end smartphones can encode a 640*480 frame within 10ms, it will be embarrassing if Jetson requires more than 66ms to do so…