Optimal time delay and jitter performance measure strategy when using gstreamer to stream camera frames

Autonomous Machines Jetson & Embedded Systems Jetson Nano
, , ,
1

Hi,

Im trying to stream video data delivered by an IMX290 connected to a board (by LI) with a nvidia-jetson nano SOM. The nvidia-jetson is running following version of L4T:
# R32 (release), REVISION: 2.1, GCID: 16294929, BOARD: t210ref, EABI: aarch64, DATE: Tue Aug 13 04:28:29 UTC 2019

Im running the following pipeline to receive the frames via RTSP/RTP:
./test-launch -p 8554 "nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM),width=1024, height=768, framerate=10/1, format=NV12 ! nvv4l2h264enc ! h264parse ! rtph264pay name=pay0 pt=96"

The pictures in the attachment describe my setup to try to evaluate the timing behaviour of the system. The pictures show a filmed (UART) terminal where the nano-board outputs its own time each 30 milli-seconds to.

cam_img_440
cam_img_4401024×768 730 KB

I have 2 question about this:

  1. It seems that the filmed “self-time” from the terminal seems to be around ~300ms before the timestamp I receive along with the RTP frame (timestamp displayed in red color). Is this delay caused by the ISP? At which point in time the timestamp I receive via RTP is taken within the nvidia-gstreamer streaming pipeline?
  2. I experience a jitter in the recording of the self-time, though the received RTP-timestamp seems to be jitter-free. How is this explainable? (the jitter is around ±10ms). Can this jitter caused by printf-stack and UART output or is the cause within the video pipeline?

Actually I want to prove that the timestamps I receive via RTP are accurate and jitter-free, do you have maybe a different performance measure strategy, with which I can achieve this?

Thanks for any help on this issue in advance.

Kind regards, Steve

2

To answer my second question about the jitter, I think my setup is a little to naive to achieve accurate results. I have only two explanations either.

  1. the time until the “eigenzeit” is outputted on TFT is not determenistic (I use teraterm on windows for outputting the eigenzeit of the board or
  2. the framerate of the camera is non-determenistic.

So would exclude 2.) for the time beeing, and try to organize a determenistic clock output to check if framerate is constant.

But with this explanation I think its unlikely that this is explaning the 300ms delay between the filmed eigenzeit and the RTP timestamp of the frame. Can someone maybe explain this delay?

Thanks in advance again.

Kind regards, Steve

4

Hi,
The latency may be from camera, encoding, network. Please try this set up:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL

To see what latency is with videotestsrc, and then try nvarguscamerasrc. By comparing the result of videotestsrc and nvarguscamerasrc, you can get latency of camera.

Also you can check glass-to-glass latency. Can refer to
Jetson Xavier AGX glass to glass latency

5

Hi @DaneLLL ,

thanks for your suggestions. Based on this, I modified clockoverlay to get current passed ms since unix epoch, so that I can compare against the RTP time which is also time in unix epoch. I get a latency (between RTP and time overlay) of 84-87 ms if I run the videotestsrc with standard framerate settings. When I decrease framerate with following pipeline:
./test-launch -p 8554 "videotestsrc is-live=1 ! video/x-raw,width=1024,height=768,framerate=10/1 ! clockoverlay valignment=4 halignment=1 ! nvvidconv ! video/x-raw(memory:NVMM),width=1024,height=768 ! nvv4l2h264enc ! h264parse ! rtph264pay name=pay0 pt=96"
I get a higher latency of around ~180ms.

If I further decrease framerate with following pipeline:
./test-launch -p 8554 "videotestsrc is-live=1 ! video/x-raw,width=1024,height=768,framerate=5/1 ! clockoverlay valignment=4 halignment=1 ! nvvidconv ! video/x-raw(memory:NVMM),width=1024,height=768 ! nvv4l2h264enc ! h264parse ! rtph264pay name=pay0 pt=96"
latency drops back to ~80ms. Iam confused :-)

Is there a connection somehow between idrinterval parameter of nvv4l2h264enc and framerate? But somehow I cant believe this because RTP timestamp should be based on buffer PTS of videotestsrc, no?

Another question I have according to the latency: In current source of nvarguscamerasrc I noticed that there is a Auxiliry data structure carrying a FrameNumber and a FrameTime. Can this data be accessed later at the rtpsink? Ideally I want to have the sensor timestamps within the stream not the buffer PTS. Also is this Auxiliry data structure also generated in nvarguscamerasrc of R32.2.1 ? (I cant verify this since the source code doesnt seem to be open).

Thanks for any info on this, Steve

6

Hi,

I’ve got an update on the latency issue I observed. I managed to eliminate all gstreamer related buffer latencies by doing the following:

I modified gstnvarguscamerasrc to retrieve the sensor timestamp like so:

static bool execute(int32_t cameraIndex,
                    int32_t cameraMode,
                    const Size2D<uint32_t>& streamSize,
                    int32_t secToRun,
                    GstNvArgusCameraSrc *src)
{
...
      iStreamSettings->setMetadataEnable(true);
...
}
bool StreamConsumer::threadExecute(GstNvArgusCameraSrc *src)
{
...
      EGLStream::IArgusCaptureMetadata *meta = Argus::interface_cast<EGLStream::IArgusCaptureMetadata>(frame);
      Argus::ICaptureMetadata* captMetaData = Argus::interface_cast<Argus::ICaptureMetadata>(meta->getMetadata());
      src->frameInfo->sensorTime = captMetaData->getSensorTimestamp();
...
}

I managed to pass this “sensorTime” as buffer metadata downstream onto the rtpsink for output. However when I then compare the timestamps (see picture)

cam_img_109
cam_img_1091024×768 447 KB

the sensor timestamp (as provided by libargus) seems to be around ~35ms ahead of its captured time. Could this be caused by TTY/monitor output latency? (The tty of the jetson should run @60FPS) or is it because of some buffering inside v4l2 driver? Can someone clarify?

Thanks for any information on this in advance.

Kind regards, Steve