Dear developers,
We use the appsrc with udp sink to deliver live video from camera to client on TX2 with the following signal processing pipeline.
------------------------------------- begin signal processing pipeline -------------------------------------
Server side:
SceneTime(t0) ->camera → (t1) Capture(at 5Hz) → BayerImgQueue → SignalProcessing–> (t2) outputYuvImgQueue → (t3)GstreamerPipeLineSendVideoToClient
GStreamer pipeline:
pipelineStr = “appsrc format=3 is-live=true ! video/x-raw,format=(string)I420, width=(int)” + std::to_string(w) + “,height=(int)” + std::to_string(h) +
" , framerate=" + std::to_string(frameRate)+ “/ 1” +
" ! identity check-imperfect-timestamp=true" +
" ! nvvidconv" +
" ! omxh264enc control-rate=2 target-bitrate=4000000 tune=zerolatency" +
" ! video/x-h264,stream-format=(string)byte-stream ! h264parse ! rtph264pay" +
" ! udpsink host=" + clientIp +" port=50000";
Client side:
UdpSrc → display (t4)
------------------------------- end signal processing pipeline -------------------------------------------
Let’s define the corresponding timestamps and time intervals as follows.
t0: time stamp on a computer screen which is captured by camera
t1: time at which a frame y is just captured
t2: time at which the frame y is processed and written into the ouputYuvImgQueue
t3: time at which the frame y is read from ouputYuvImgQueue by need_data_callback()
t4:time stamp on receiver side screen
dt1 = t2-t1
dt2 = t3-t2
dt3 = t4-t0 (glass-to-glass latency)
Our experiments show that the glass-to-glass latency strongly depends on the frameRate parameter in video/x-raw cap. For example
frameRate dt1(ms) dt2 range(ms) max dt3(ms)
5 25 [700,850] ~1290
6 25 [270,300] ~740
7 25 [200,260] ~690
10 25 [150,200] ~630
15 25 [110,120] ~550
20 25 [92,100] ~530
30 25 [75,79] ~500
Note: when there is no new image in outputYuvImgQueue and need_data_callback() is called, we just repeat the previous frame.
My Question is why the frameRate parameter affects dt2 so much? To void this, what kind of parameter shall we tune?
Thanks,