Extreme amounts of artifacts in video when running a GStreamer pipeline as a service through systemd

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hello. I have some code that sets up a GStreamer pipeline to stream video over RTSP from a CSI/MIPI camera using nvarguscamerasrc. The code uses nvv4l2h264enc to H.264 encode the video and then it sets up an RTSP stream using GstRTSPMediaFactory and rtph264pay.

We are using a Jetson TX2 on a Spacely carrier board from ConnectTech. The camera is LI-IMX274-MIPI-M12 from Leopard Imaging.

The code to regenerate the issue can be found here: https://github.com/oysteinskotheim/nvidia-rtsp-startup-artifacts

To build and run the code to reproduce the issue:

mkdir build && cd build
cmake ..
make
sudo make install

Then enable it as a startup service through systemd: sudo systemctl enable rtsp-server and reboot.

To connect to the RTSP stream type gst-launch-1.0 playbin uri=rtsp://<ip>/video0 uridecodebin0::source::latency=0

On one out of every few reboots, there will be extreme amounts of artifacts in the video that appears.

Example screenshot when the video stream looks fine:

image
image1274×744 67.5 KB

Example screenshot from the video stream when the issue appears:

image
image1274×744 93 KB

After restarting the rtsp_server process, the video stream looks fine again.

It is worth to mention that I have not been able to reproduce the issue when starting the rtsp server normally from the command line. It only happens sporadically (1 out of 3 or 4 times) when running it via systemd.

Can I please get some support on what may be the root cause of this issue? We are using similar code in an embedded product, but every few times the system is started the video stream is so full of artifacts that it is unusable.

Best regards,
-Øystein

3

The artifacts are caused by packet lost during RTSP transferring.

4

Fiona: The GStreamer pipeline seems to sometimes end up in a bad state where the video is stuttering and full of artifacts even when connecting to the stream locally (or from a computer connected to the same Gigabit Ethernet switch). When the RTSP server is restarted the problem disappears and the video is smooth and without artifacts. So I don’t think this problem occurs due to packet loss. It occurs one out of every few times the RTSP server is started from systemd. I’m wondering if it could be a bug or a race condition in the camera driver or one of the GStreamer plugins.

5

From the image you showing, it is a typical packet lost video decoding result. You can debug your system to verify your conjecture.

6

The problem does not occur due to network issues. It occurs even when I try to view the stream on localhost. The video stream looks fine when the rtsp server is launched normally on the command line. The problem only occurs when the RTSP server is started automatically on startup via systemd. Then the video is stuttering and full of artifacts and it remains like that until the RTSP server process is restarted.

7

Even with “localhost”, packet lost is possible. The tcp/ip protocol stack is purely software (only physical layer is HW).

8

How can we proceed to resolve this issue? As I said, the issue appears even on localhost, so the issue is not caused by bandwidth/network issues. The GStreamer pipeline or camera driver seems to end up in a bad state every few times when it is started by systemd, where the video is completely stuttered and full of artifacts. It starts up in a bad state and remains like that until the process is restarted. After the process is restarted, the video stream is perfectly fine again and we see no issues until the system is rebooted.

9

You need to check whether tcp/ip stack and other dependencies are ready before you start deepstream app in systemd.

10

This is what the rtsp-server.service file looks like:

## RTSP server

[Unit]
Description=RTSP server
After=nvargus-daemon.service
After=network-online.target

[Service]
Type=simple
Restart=always
RestartSec=3
ExecStart=/usr/local/bin/rtsp_server

[Install]
WantedBy=multi-user.target

Is there anything else it needs to be started after?

I also tried to add a 10 seconds delay:

# Delay start with 10 seconds
ExecStartPre=/bin/sleep 10
ExecStart=/usr/local/bin/rtsp_server

but the problem still persists.

11

To identify who caused the packet loss, you don’t need to run deepsteam app and the rtsp server. Just use some packet loss tool with systemd is enough.(e.g. iperf)

It has nothing to do with deepstream.

Or you can try to set smaller “iframeinterval” value with nvv4l2h264encoder to check whether it can help the case.

12

The Jetson TX2 and my computer are connected to the same Gigabit Ethernet switch. I ran a test using iperf:

$ iperf -c 10.10.0.99
------------------------------------------------------------
Client connecting to 10.10.0.99, TCP port 5001
TCP window size:  512 KByte (default)
------------------------------------------------------------
[  3] local 10.10.0.149 port 2966 connected with 10.10.0.99 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes   938 Mbits/sec

So I don’t think there are issues concerning the bandwidth.

13

Please check your board by yourself. It is not deepstream issue.

14

I have tried the code on at least 5 different boards. They all exhibit the same behavior.

Below you can see two screen captures. In the first one, I connect to the RTSP server (see code for reproducing the issue on GitHub) right after startup where the video stream is defective and full of artifacts.

Then another screen capture after restarting the server process with sudo systemctl restart rtsp-server and connecting to the RTSP server again. As you can see, the video stream now looks fine.

15

Packet lost is not caused by deepstream. Please check by yourself.

16

I have already checked with iperf that I have no bandwidth issues. How do you want me to check it?

17

The network stack and rtsp stack is not provided by deepstream.

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.