Jetson Nano UDP Stream to OpenCV Latency and Frame Drop

Autonomous Machines Jetson & Embedded Systems Jetson Nano
, ,
1

Hardware:

  • NVIDIA Jetson Nano Developer Kit with JetPack 4.6.1 on MicroSD Card
  • Raspberry Pi Camera v2

Background: I am writing a program for a system that needs to be able to perform object detection on live camera feed, subsequently go into suspend mode to save power, and resume the object detection program immediately upon waking up. I’ve found that the video capturing and video analysis need to be split into two separate programs because GStreamer does not support entering suspend/resume in runtime while the GStreamer pipeline is running.

Problem: I have decided to first try UDP for transmitting video from the command line to a Python program to test having two separate programs for video capturing and video analysis. The results are that there is ~300ms of latency on the receiving end and massive amounts of frame drops. Below is the code:

Sender:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=60/1' ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0

Receiver:

import cv2
import gi
gi.require_version("Gst", "1.0")
from gi.repository import Gst

Gst.init(None)

pipeline = (
    "udpsrc port=5000 ! application/x-rtp,encoding-name=H264,payload=96 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! appsink"
)

cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)

while True:
    ret, frame = cap.read()
    if not ret:
        break

    cv2.imshow("UDP Video Stream", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

Note: There is litte to no frame drops and ~30ms latency when the receiver is ran with this code:

gst-launch-1.0 udpsrc port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! avdec_h264 ! xvimagesink sync=0

Questions:

  • Is this a problem with the speed of Python? Would C or C++ be a faster alternative?
  • Is UDP the best protocol for this use case? Would TCP, RTSP, or something else be better?
  • Are there modifications I could make to the Python code to solve the latency and frame drop issues?

Any help is much appreciated.

3

Hi,
Please try the setup:

Jetson Nano FAQ
Q: Is there an example for running UDP streaming?

And check the latency. We can compare videotestsrc and nvarguscamerasrc to identify how much latency is from the camera source.

4

Test 1: videotestsrc
Upon testing videotestsrc, the result is ~30ms latency with the following commands:
Sender:

gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=1280,height=720 ! timeoverlay valignment=4 halignment=1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0

Receiver:

gst-launch-1.0 udpsrc port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! avdec_h264 ! xvimagesink sync=0

Test 2: nvarguscamerasrc
Upon testing nvarguscamerasrc, the result was ~30ms as well, however, I found that when I put the framerate at 60/1, the latency was much higher than at 30/1 framerate. Below are the commands I tested:

Sender (framerate=30/1):

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1' ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0

Sender (framerate=60/1):

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=60/1' ! nvvidconv flip-method=2 ! 'video/x-raw(memory:NVMM),width=1280,height=720' ! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 ! h264parse ! rtph264pay ! udpsink host=127.0.0.1 port=5000 sync=0 t. ! queue ! nvegltransform ! nveglglessink sync=0

Receiver:

gst-launch-1.0 udpsrc port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! avdec_h264 ! xvimagesink sync=0

The result of Test 2 is strange to me because 1280x720 30fps is not a valid sensor mode. In both Sender tests in Test 2, where framerate=30/1 and framerate=60/1, the camera is still put into camera mode 5 with 120fps. I am unable to put the camera into camera mode 4.

GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005
6

Hi,
Please run the command and check the frame rate:

gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

The camera can be in mode 5 and generate frame data in 30fps.

And for 60fps, please try idrinterval=30 or idrinterval=60. IDR frames are with more data bits and instant data rate is high. For 60fps + idrinterval=15, there are 4 IDR frames per second and this may trigger some latency. Please try larger interval to see if there is improvement.

7

This command does prove that the framerate is being set as intended.

It also appears that increasing the idrinterval does improve upon the 60fps latency slightly, although the latency is still much greater than when framerate=30/1.

I’d like to address my original questions about receiving and displaying the video feed in a Python program with low latency and minimal dropped frames in order to perform object detection on the real-time video stream. What can be done differently to achieve this?

8

Hi,
We think the performance may be better in C/C++ code. Since generally there is python-bindind layer for python code. If you would like to run deep learning inference, may consider use DeepStream SDK. Please refer to documents:
https://docs.nvidia.com/metropolis/deepstream-archive.html

The latest version for Jetson Nano is 6.0.1. Please take a look and try the samples.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.