Hello,
I have been trying to get a low latency video stream from camera to another device via UDP.
After some testing of 2 cameras (IMX219 and OV9281), capturing the image and displaying it on the jetson hdmi output (60Hz display) without streaming it via udp, I got these results:
IMX219 1920x1080 60fps 100-120ms
gst-launch-1.0 nvarguscamerasrc ! "video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12, framerate=30/1" ! nvvidconv ! fpsdisplaysink sync=false async=false
IMX219 1920x1080 60fps no ISP 80-100ms
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080 --set-ctrl sensor_mode=2 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=50 gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! nvvidconv ! fpsdisplaysink sync=false
IMX219 1280x720 60fps 65-80ms
gst-launch-1.0 nvarguscamerasrc ! "video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=60/1" ! nvvidconv ! fpsdisplaysink sync=false async=false
IMX219 1280x720 60fps no ISP 65ms
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=720 --set-ctrl sensor_mode=5 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=200 gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! nvvidconv ! fpsdisplaysink sync=false
OV9281 1280x800 60fps 85-100ms limited to 34fps?
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=800 --set-ctrl sensor_mode=0 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=200 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=800 ! videoconvert ! fpsdisplaysink sync=false & sleep 2 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=59 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=60
OV9281 640x400 60fps 50-65ms
v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=400 --set-ctrl sensor_mode=2 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=500 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=640,height=400 ! videoconvert ! fpsdisplaysink sync=false & sleep 2 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=59 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=60
OV9281 640x400 60fps io-mode=2 50-65ms, dips to 35ms
v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=400 --set-ctrl sensor_mode=2 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=500 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! video/x-raw,width=640,height=400 ! videoconvert ! fpsdisplaysink sync=false & sleep 2 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=59 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=60
OV9281 1280x800 60fps 65ms, with H264 encoding and decoding
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=800 --set-ctrl sensor_mode=0 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=200 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! video/x-raw,width=1280,height=800 ! nvvidconv ! nvv4l2h264enc bitrate=2000000 ! h264parse ! rtph264pay ! application/x-rtp, encoding-name=H264, payload=96 ! rtph264depay ! h264parse ! avdec_h264 ! fpsdisplaysink sync=false & sleep 2 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=59 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=60
OV9281 640x400 60fps 35-65ms, with H264 encoding and decoding
v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=400 --set-ctrl sensor_mode=2 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=500 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! video/x-raw,width=640,height=400 ! nvvidconv ! nvv4l2h264enc bitrate=2000000 ! h264parse ! rtph264pay ! rtph264depay ! h264parse ! avdec_h264 ! fpsdisplaysink sync=false & sleep 2 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=59 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=60
OV9281 640x400 60fps with H264 encoding and decoding without h264parse causes the latency to be 150ms
v4l2-ctl -d /dev/video0 --set-fmt-video=width=640,height=400 --set-ctrl sensor_mode=2 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=500 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! video/x-raw,width=640,height=400 ! nvvidconv ! nvv4l2h264enc bitrate=2000000 ! avdec_h264 ! fpsdisplaysink sync=false & sleep 2 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=59 v4l2-ctl -d /dev/video0 --set-ctrl frame_rate=60
Im wondering if it can be improved, or find out whats the bottleneck.
Since im testing on 60Hz display I know thats also going to be a factor.
Why when switching to higher resolutions the latency increases, even when not using the encoder and even then the encoding doesnt take more then 10ms to encode the image according to gst-shark tracing.
Setup:
Orin NX 16GB
JetPack 5.1.2