UDP-Raw Stream Issue On Nvidia Jetson Devices

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hello fellow developers,
we are working on a camera system which dishes out ~ 120Mbits/s raw data as UDP/RTP stream. We have connected the camera system and receiving devices P2P with a single ethernet cable. When we try to pick up that stream from a regular device(laptop or desktop) the video is flawless. But as we try to receive the stream from a Jetson device, the video suffers heavily from interlacing especially when there are moving objects.
Example image:

UDP-RAW-cropped

In all cases we have used the base Gstreamer pipeline below to recieve the stream:

gst-launch-1.0 -v udpsrc uri=udp://192.168.1.103:5555 caps="application/x-rtp, 
media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, 
sampling=(string)YCbCr-4:2:2, depth=(string)8, width=(string)1920, height=(string)1080"  
! rtpvrawdepay ! queue ! videoconvert ! xvimagesink

We have confirmed that the issue only appears when Jetson devices try to handle RAW data. By streaming laptop camera to Jetson, first RAW then JPEG encoded, we have observed the same issue on RAW stream but not on JPEG encoded one.
Laptop Camera Stream Recieved On Jetson TX2:

  • RAW ~ 36 Mbits/s β†’ Interlacing
  • JPEG encoded ~ 12 Mbits/s β†’ No Interlacing

Laptop Camera Stream Recieved On Jetson Nano:

  • RAW ~ 36 Mbits/s β†’ Interlacing
  • JPEG encoded ~ 12 Mbits/s β†’ No Interlacing

Laptop Camera Stream Recieved On Desktop PC:

  • RAW ~ 36 Mbits/s β†’ No Interlacing
  • JPEG encoded ~ 12 Mbits/s β†’ No Interlacing

We have also experimented with gstreamer plugins such as rtpjitterbuffer, deinterlace and tried changing frame format(YCbCr-4:2:2, YCbCr-4:2:0, RGBA, RGB) on streaming device. But no avail. Our suspicion is that the gstreamer plugin rtpvrawdepay does not function correctly on Jetson devices. Which causes some udp/rtp packets to be dropped or misordered.
Could you please assist ?
Setup:

  • Jetson TX2
  • Jetpack 4.4
  • Gstreamer 1.14.5
  • Jetson Nano
  • Jetpack 4.4
  • Gstreamer 1.14.5
3 Likes
3

Hi,
Please execute sudo nvpmodel -m 0 and sudo jetson_clocks. It is the mode with max performance. You may refer to detail in document:
https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fpower_management_tx2_32.html%23

And you may try to encode into h264 stream. There is an example:

We are deprecating omx plugins. IF you would like to try, please use nvv4l2h264enc and nvv4l2decoder instead of omxh264enc and omxh264dec.

4

Hi DaneLLL,
in all of our tests the devices were in max performance mode( nvpmodel -m 0 ). We have also concluded that jetson clocks has no effect on our situation. Issue still persists.
Working with raw data is one of our limitations so encoding with h264 is not an option at the moment. It also shouldn’t be necessary since our bandwith need is well within bandwith limits. Jetson boards offer a bandwidth of 1Gbits/s and our data is a mere 120Mbits/s. It even occurs when streaming from laptop camera to jetson (36Mbit/s). I am really struggling to find out what is causing this distortion while receiving the raw video.
To reproduce issue with a laptop camera and peer2peer connection:

Laptop side

gst-launch-1.0 -v v4l2src ! video/x-raw, width=1280, height=720 ! rtpvrawpay ! udpsink host=192.168.1.103 port=5005

Jetson side

gst-launch-1.0 -v udpsrc uri=udp://192.168.1.103:5005 caps= "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:2, depth=(string)8, width=(string)1280, height=(string)720" ! rtpvrawdepay ! videoconvert ! xvimagesink
5

Hi,
Setting sync=0 might help. Please try

... ! xvimagesink sync=0
6

Hello again Dane,
we have experimented with several properties from following elements:

  • xvimagesink β†’ sync, async, max-lateness
  • videoconvert β†’ primaries-mode, β€˜dither’
  • deinterlace β†’ mode, method, tff, locking
  • interlace β†’ field-pattern
  • udpsrc β†’ blocksize, num-buffers, buffer-size, caps (sampling,depth)
  • rtpjitterbuffer β†’ latency, drop-on-latency, mode

nothing above has solved our case. We have also tested that when we use Jetson devices as streamer and our laptop as receiver there are no problems. So Jetson devices can stream RAW but cannot receive RAW stream correctly?

Test conditions;

  • Devices: Jetson Nano 4GB developerkit Jetpack 4.4, Jetson TX2 Developerkit Jetpack 4.4, Dell G3 p75f Ubuntu 18.04
  • Connection: Peer 2 peer using single ethernet cable (Jetson IP β†’ 192.168.1.103 , Laptop Ubuntu 18.04 IP β†’ 192.168.1.104)
  • Performance Mode: Jetson-Clocks enabled, Max Performance Mode (MAXN)
  • Gstreamer Version: 1.14.5 (same on both devices)
  • Cameras: Onboard Webcam, CSI Raspberry camera (Jetson)
  • Format: UDP/RTP , Raw

Results:

  • Jetson to Laptop β†’ No issues on video
  • Laptop to Jetson β†’ Line by line tearing on video horizontally

Could you please try to reproduce the issue Dane?

7

HΔ° @DaneLLL
We are still struggling with the same issue here. How are the results on your end? Did you manage to reproduce the issue? Is there anything more I can provide for this issue to be tested? Please let me know. Thank you for your time.

8

Hi,
Sending raw data in UDP streaming requires high CPU usage and network bandwidth. This may hit system limitation. Since there is hardware encoder/decoder in TX2, we would suggest compress the data into h264/h265 stream. This looks to be a more reliable usecase.

11

For anyone reaching out there for the same, you may also try increasing kernel recv socket buffer max size such as in Raw data video not stable by gstreamer rtpvrawpay - #8 by Honey_Patouceul

1 Like