Using RTSP encode (with H264) and stream images at device memory on Xavier

Hello Jetson team and developers,

Our Problem: To encode and transmit real-time images processed (by NPP) and stored in device memory ( allocated by cudaMalloc()) at Xavier with H264 codec format.

An approach tried: We copy the image from device to host and using FFMPEG RTSP, it works but too slow.

Approaches to investigate:
A. Using gstreamer RTSP directly on device memory, and take its hardware encoding advantages.
B. RTSP using DeepStream SDK (we guess the performance will be similar with that in A, is this guess correct? )

For approach A, I found an example at
https://github.com/GStreamer/gst-rtsp-server/blob/master/examples/test-launch.c
But it transmit a video file with g_main_loop, are there any examples to use a for-loop to do the frame-by- frame process?

I also found there is a discuss on how to directly access cuda memory by GstBuffer at here
https://devtalk.nvidia.com/default/topic/1023932/jetson-tx2/process-multiple-gstreamer-buffers-directly-allocated-in-cuda-memory/post/5209904/#5209904

To be clear, i do not want to change my current architecture under the g-streamer hood, i only want to do rtsp for a frame whenever it’s produced by another thread.

Thank you,

-shunguang

Hi,
We suggest you go to http://gstreamer-devel.966125.n4.nabble.com/ and get suggestions for your usecase.

Once you have a full pipeline, you can replace x264enc with omxh264enc/nvv4l2h264enc to leverage Xavier HW encoder.

Some rtsp cases(not 100% same as yours though) for your reference:
https://devtalk.nvidia.com/default/topic/1018689/jetson-tx2/vlc-playing-gstreamer-flow/post/5187270/#5187270
https://devtalk.nvidia.com/default/topic/1014789/jetson-tx1/-the-cpu-usage-cannot-down-use-cuda-decode-/post/5188538/#5188538
https://devtalk.nvidia.com/default/topic/1043770/jetson-tx2/problems-minimizing-latency-and-maximizing-quality-for-rtsp-and-mpeg-ts-/post/5295828/#5295828

Thanks DaneLLL, Once I have an answer, will post it.

Hi DaneLLL,

I posted my question on gstreamer-devel message board, unfortunately did get any response. Then I spent some time on reading GStreamer Application Development Manual (1.10.1), and have the feeling that there is a steep learning curve if going through that approach.

On the other hand, when look at the FFmpeg Video Codec SDK at:
https://developer.nvidia.com/ffmpeg
it seems that FFMPEG already has the ability to employ Xavier HW encoder. If so i just need to compile the latest FFMPEG. Please confirm if this is true.

Thank you so much!

Hi,
Video Codec SDK is for desktop GPUs. For Jetson TX1/TX2/Xavier/Nano, we do not support HW acceleration in ffmpeg.

Thanks, DaneLLL!

Actually i found test-appsrc.c from
https://github.com/GStreamer/gst-rtsp-server/blob/master/examples/test-appsrc.c
is a good start point to tackle this problem.

First in test_appsrc(), we need to change parameters in gst_rtsp_media_factory_set_launch () from
β€œ( appsrc name=mysrc ! videoconvert ! x264enc ! rtph264pay name=pay0 pt=96 )”
into
β€œ( appsrc name=mysrc ! omxh264enc ! rtph264pay name=pay0 pt=96 )”

Then in media_configure() change format from β€œRGB16” into β€œI420”.

Next in need_data(), try to create a YUV420 image and use gst_buffer_fill() copy it into GstBuffer *buffer.

Once this prototype works, we can further modify optimize need_data() to accept images from an outside queue allocated in device.

Again, thank you for the great suggestions.

1 Like