Appsrc ! video/x-raw, frame rate and latency

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Dear developers,

We use the appsrc with udp sink to deliver live video from camera to client on TX2 with the following signal processing pipeline.

------------------------------------- begin signal processing pipeline -------------------------------------
Server side:
SceneTime(t0) ->camera → (t1) Capture(at 5Hz) → BayerImgQueue → SignalProcessing–> (t2) outputYuvImgQueue → (t3)GstreamerPipeLineSendVideoToClient

GStreamer pipeline:
pipelineStr = “appsrc format=3 is-live=true ! video/x-raw,format=(string)I420, width=(int)” + std::to_string(w) + “,height=(int)” + std::to_string(h) +
" , framerate=" + std::to_string(frameRate)+ “/ 1” +
" ! identity check-imperfect-timestamp=true" +
" ! nvvidconv" +
" ! omxh264enc control-rate=2 target-bitrate=4000000 tune=zerolatency" +
" ! video/x-h264,stream-format=(string)byte-stream ! h264parse ! rtph264pay" +
" ! udpsink host=" + clientIp +" port=50000";

Client side:
UdpSrc → display (t4)
------------------------------- end signal processing pipeline -------------------------------------------

Let’s define the corresponding timestamps and time intervals as follows.
t0: time stamp on a computer screen which is captured by camera
t1: time at which a frame y is just captured
t2: time at which the frame y is processed and written into the ouputYuvImgQueue
t3: time at which the frame y is read from ouputYuvImgQueue by need_data_callback()
t4:time stamp on receiver side screen
dt1 = t2-t1
dt2 = t3-t2
dt3 = t4-t0 (glass-to-glass latency)

Our experiments show that the glass-to-glass latency strongly depends on the frameRate parameter in video/x-raw cap. For example

frameRate   dt1(ms)   dt2 range(ms)	 max dt3(ms)
5           25        [700,850]         ~1290
6	        25	      [270,300]	         ~740 		
7	        25	      [200,260]          ~690			
10          25        [150,200]          ~630    
15          25        [110,120]          ~550
20          25        [92,100]           ~530
30          25        [75,79]            ~500

Note: when there is no new image in outputYuvImgQueue and need_data_callback() is called, we just repeat the previous frame.

My Question is why the frameRate parameter affects dt2 so much? To void this, what kind of parameter shall we tune?

Thanks,

3

Hi,
Please refer to the cases of UDP streaming:
Jetson 4k Encoding -> Decoding Pipeline and latency - #11 by DaneLLL
Delay in video - #5 by DaneLLL

If your source is in steady rate, we suggest disable synchronization mechanism in gstreamer by setting sync=0 in client pipeline for a try.

5

Here are the pipelines with acceptable latency for our application:

Server side on Tx2

	const int w = m_videoInfo.w;
	const int h = m_videoInfo.h;
    const uint64_t minLatency = (m_videoInfo.frame_rate.den * 1000000000)/m_videoInfo.frame_rate.num;
    const uint64_t maxLatency = 1000000000;   //1sec
	const int bFrms = m_videoInfo.frame_rate.num/m_videoInfo.frame_rate.den;
	const std::string clientIp = ipConvertNum2Str(m_clientInfo.ss_.clientIp);  //"127.0.0.1";
	const int br_Kbps = m_videoInfo.bitrate_Kbps;
	const int br_bps = 1000*m_videoInfo.bitrate_Kbps;
	const string liveStr = (m_videoInfo.isLive) ? "true" : "false";
	const string syncStr = (m_videoInfo.isLive) ? "false" : "true";
    std::string  pipelineStr = "appsrc name=" + m_appsrcName + " format=3 is-live=" + liveStr;
	pipelineStr += " ! video/x-raw,format=(string)I420,width=(int)" + std::to_string(w) + ",height=(int)" + std::to_string(h) + 
        " , framerate=(fraction)" + std::to_string(m_videoInfo.frame_rate.num)+ "/" + std::to_string(m_videoInfo.frame_rate.den) + 
        " ! identity check-imperfect-timestamp=true" +
        " ! nvvidconv" +
		" ! omxh264enc control-rate=2 bitrate="+std::to_string(br_bps) +" low-latency=1 maxperf-enable=true" +  
		" ! video/x-h264,stream-format=(string)byte-stream ! h264parse ! rtph264pay" +
        " ! udpsink host=" + clientIp +" port=50000 sync=" + syncStr + " async=false";

Client sider on Andriod
udpsrc port=50000 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=96 ! rtpjitterbuffer mode=0 drop-on-latency=true latency=200 max-dropout-time=400 max-misorder-time=200 ! rtph264depay ! h264parse ! decodebin ! autovideosink sync=false async=false

2 Likes