Appsrc ! video/x-raw, frame rate and latency

Dear developers,

We use the appsrc with udp sink to deliver live video from camera to client on TX2 with the following signal processing pipeline.

------------------------------------- begin signal processing pipeline -------------------------------------
Server side:
SceneTime(t0) ->camera → (t1) Capture(at 5Hz) → BayerImgQueue → SignalProcessing–> (t2) outputYuvImgQueue → (t3)GstreamerPipeLineSendVideoToClient

GStreamer pipeline:
pipelineStr = “appsrc format=3 is-live=true ! video/x-raw,format=(string)I420, width=(int)” + std::to_string(w) + “,height=(int)” + std::to_string(h) +
" , framerate=" + std::to_string(frameRate)+ “/ 1” +
" ! identity check-imperfect-timestamp=true" +
" ! nvvidconv" +
" ! omxh264enc control-rate=2 target-bitrate=4000000 tune=zerolatency" +
" ! video/x-h264,stream-format=(string)byte-stream ! h264parse ! rtph264pay" +
" ! udpsink host=" + clientIp +" port=50000";

Client side:
UdpSrc → display (t4)
------------------------------- end signal processing pipeline -------------------------------------------

Let’s define the corresponding timestamps and time intervals as follows.
t0: time stamp on a computer screen which is captured by camera
t1: time at which a frame y is just captured
t2: time at which the frame y is processed and written into the ouputYuvImgQueue
t3: time at which the frame y is read from ouputYuvImgQueue by need_data_callback()
t4:time stamp on receiver side screen
dt1 = t2-t1
dt2 = t3-t2
dt3 = t4-t0 (glass-to-glass latency)

Our experiments show that the glass-to-glass latency strongly depends on the frameRate parameter in video/x-raw cap. For example

frameRate   dt1(ms)   dt2 range(ms)	 max dt3(ms)
5           25        [700,850]         ~1290
6	        25	      [270,300]	         ~740 		
7	        25	      [200,260]          ~690			
10          25        [150,200]          ~630    
15          25        [110,120]          ~550
20          25        [92,100]           ~530
30          25        [75,79]            ~500

Note: when there is no new image in outputYuvImgQueue and need_data_callback() is called, we just repeat the previous frame.

My Question is why the frameRate parameter affects dt2 so much? To void this, what kind of parameter shall we tune?

Thanks,

Hi,
Please refer to the cases of UDP streaming:
Jetson 4k Encoding -> Decoding Pipeline and latency - #11 by DaneLLL
Delay in video - #5 by DaneLLL

If your source is in steady rate, we suggest disable synchronization mechanism in gstreamer by setting sync=0 in client pipeline for a try.

Here are the pipelines with acceptable latency for our application:

Server side on Tx2

	const int w = m_videoInfo.w;
	const int h = m_videoInfo.h;
    const uint64_t minLatency = (m_videoInfo.frame_rate.den * 1000000000)/m_videoInfo.frame_rate.num;
    const uint64_t maxLatency = 1000000000;   //1sec
	const int bFrms = m_videoInfo.frame_rate.num/m_videoInfo.frame_rate.den;
	const std::string clientIp = ipConvertNum2Str(m_clientInfo.ss_.clientIp);  //"127.0.0.1";
	const int br_Kbps = m_videoInfo.bitrate_Kbps;
	const int br_bps = 1000*m_videoInfo.bitrate_Kbps;
	const string liveStr = (m_videoInfo.isLive) ? "true" : "false";
	const string syncStr = (m_videoInfo.isLive) ? "false" : "true";
    std::string  pipelineStr = "appsrc name=" + m_appsrcName + " format=3 is-live=" + liveStr;
	pipelineStr += " ! video/x-raw,format=(string)I420,width=(int)" + std::to_string(w) + ",height=(int)" + std::to_string(h) + 
        " , framerate=(fraction)" + std::to_string(m_videoInfo.frame_rate.num)+ "/" + std::to_string(m_videoInfo.frame_rate.den) + 
        " ! identity check-imperfect-timestamp=true" +
        " ! nvvidconv" +
		" ! omxh264enc control-rate=2 bitrate="+std::to_string(br_bps) +" low-latency=1 maxperf-enable=true" +  
		" ! video/x-h264,stream-format=(string)byte-stream ! h264parse ! rtph264pay" +
        " ! udpsink host=" + clientIp +" port=50000 sync=" + syncStr + " async=false";

Client sider on Andriod
udpsrc port=50000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96 ! rtpjitterbuffer mode=0 drop-on-latency=true latency=200 max-dropout-time=400 max-misorder-time=200 ! rtph264depay ! h264parse ! decodebin ! autovideosink sync=false async=false

2 Likes