Failed to stream GPU encoded rtsp-server using gstreamer

[manishkr]

Hello
I’m trying to host h264/h265 GPU encoded rtsp from my inference pipeline using nvv4l2{codec}enc/nv{codec}enc with gstreamer but unable to launch stream

so far only x264enc/x265enc plugin based pipeline is working our case

appsrc name=source is-live=true block=true format=GST_FORMAT_TIME caps=video/x-raw,format=BGR,width=1280,height=720,framerate=15/1 ! videoconvert ! video/x-raw,format=I420 ! x264enc speed-preset=ultrafast tune=zerolatency ! rtph264pay config-interval=1 name=pay0 pt=96

when we tried to run nv{codec}enc based pipeline

appsrc name=source caps=video/x-raw,format=BGR,width=1280,height=720,framerate=15/1 ! videoconvert ! video/x-raw,format=I420 ! nvvidconv ! nvh265enc ! h265parse ! rtph265pay config-interval=1 name=pay0 pt=96

we got following warnings through GST_DEBUG=3

0:00:22.479802733 1 0x7f5bcc007330 WARN rtspmedia rtsp-media.c:4623:gst_rtsp_media_suspend: media 0x7f5b60786d80 was not prepared
0:00:22.483181366 1 0x7f5bcc007330 WARN rtspstream rtsp-stream.c:5599:gst_rtsp_stream_query_position:GstRTSPStream@0x7f5b608d2df0 Couldn’t obtain postion: position query failed

when we tried to run nvv4l2{codec}enc based pipeline

appsrc name=source is-live=true block=true format=GST_FORMAT_TIME ! videoconvert ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=15/1,format=NV12 ! nvv4l2h265enc ! h265parse ! rtph265pay config-interval=1 name=pay0 pt=96

we got following logs through GST_DEBUG-=3

0:00:05.423771134 1 0x7f43640082d0 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop: error: Internal data stream error.
0:00:05.423816489 1 0x7f43640082d0 WARN basesrc gstbasesrc.c:3127:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4)
0:00:05.424040527 1 0x7f4364007b00 WARN rtspmedia rtsp-media.c:3268:default_handle_message: 0x7f42f0019d20: got error Internal data stream error. (…/libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:media-pipeline/GstPipeline:pipeline1/GstAppSrc:source:
streaming stopped, reason not-negotiated (-4))
0:00:05.424136203 1 0x7f4364007330 WARN rtspmedia rtsp-media.c:3594:wait_preroll: failed to preroll pipeline
0:00:05.424179224 1 0x7f4364007330 WARN rtspmedia rtsp-media.c:3964:gst_rtsp_media_prepare: failed to preroll pipeline
0:00:05.425189328 1 0x7f4364007330 ERROR rtspclient rtsp-client.c:1087:find_media: client 0x7f436c510c30: can’t prepare media
0:00:05.425544277 1 0x7f4364007330 ERROR rtspclient rtsp-client.c:3376:handle_describe_request: client 0x7f436c510c30: no media

I would appreciate any help.
Thanks
Manish

[DaneLLL]

Hi,
Please try the command and see if it works:

appsrc name=source caps=video/x-raw,format=BGR,width=1280,height=720,framerate=15/1 ! videoconvert ! video/x-raw,format=I420 ! nvvidconv ! nvv4l2h265enc ! h265parse ! rtph265pay config-interval=1 name=pay0 pt=96

On Jetson platforms, we have nvv4l2h264enc and nvv4l2h265enc plugins for hardware encoding.

[manishkr]

@DaneLLL

video/x-raw,format=I420 ! nvvidconv

Hi
I tried running above pipeline on RTX A4000

appsrc name=source is-live=true block=true format=GST_FORMAT_TIME caps=video/x-raw,format=BGR,width=1280,height=720,framerate=15/1 ! videoconvert ! video/x-raw,format=I420 ! nvvideoconvert ! nvv4l2h265enc bitrate=600 ! h265parse ! rtph265pay config-interval=1 name=pay0 pt=96

getting these error logs now

0:00:26.658603717 1 0x7f02f0008000 WARN v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:nvv4l2h265enc0:src Could not probe maximum capture size for pixelformat H265
ENC_CTX(0x7f0218017a90) Error in initializing nvenc context
0:00:26.659470952 1 0x7f02f0008060 ERROR v4l2allocator gstv4l2allocator.c:784:gst_v4l2_allocator_start:nvv4l2h265enc0:pool:src:allocator error requesting 2 buffers: Cannot allocate memory
0:00:26.659479338 1 0x7f02f0008060 ERROR v4l2bufferpool gstv4l2bufferpool.c:1217:gst_v4l2_buffer_pool_start:nvv4l2h265enc0:pool:src we received 0 buffer from device ‘/dev/v4l2-nvenc’, we want at least 2
0:00:26.659482861 1 0x7f02f0008060 ERROR bufferpool gstbufferpool.c:572:gst_buffer_pool_set_active:nvv4l2h265enc0:pool:src start failed
0:00:26.659485561 1 0x7f02f0008060 WARN v4l2videoenc gstv4l2videoenc.c:1053:gst_v4l2_video_enc_set_format: Could not activate capture buffer pool.
0:00:26.659491692 1 0x7f02f0008060 WARN videoencoder gstvideoencoder.c:778:gst_video_encoder_setcaps: rejected caps video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)15/1, format=(string)I420, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:26.659542646 1 0x7f02f0008060 WARN v4l2 gstv4l2object.c:3565:gst_v4l2_object_set_format_full: error: Device is in streaming mode
0:00:26.659554408 1 0x7f02f0008060 WARN videoencoder gstvideoencoder.c:778:gst_video_encoder_setcaps: rejected caps video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)15/1, format=(string)I420, block-linear=(boolean)false, nvbuf-memory-type=(string)nvbuf-mem-cuda-device, gpu-id=(int)0
0:00:26.659559069 1 0x7f02f0008060 WARN GST_PADS gstpad.c:4351:gst_pad_peer_query:nvvideoconvert1:src could not send sticky events
0:00:26.659627675 1 0x7f02f0008000 WARN rtspmedia rtsp-media.c:3268:default_handle_message: 0x7f028001aab0: got error Could not get/set settings from/on resource. (gstv4l2object.c(3565): gst_v4l2_object_set_format_full (): /GstPipeline:media-pipeline/GstPipeline:pipeline1/nvv4l2h265enc:nvv4l2h265enc0:
Device is in streaming mode)
0:00:26.659640854 1 0x7f02f00075b0 WARN rtspmedia rtsp-media.c:3594:wait_preroll: failed to preroll pipeline
0:00:26.659650375 1 0x7f02f00075b0 WARN rtspmedia rtsp-media.c:3964:gst_rtsp_media_prepare: failed to preroll pipeline
ERROR:tapvision.utils.rtsp_server:
0:00:26.661667375 1 0x7f02f00075b0 ERROR rtspclient rtsp-client.c:1087:find_media: client 0x7f02f4266790: can’t prepare media
0:00:26.661759525 1 0x7f02f00075b0 ERROR rtspclient rtsp-client.c:3376:handle_describe_request: client 0x7f02f4266790: no media

[DaneLLL]

Hi,
It seems like you are not using Xavier NX. Please share which platform you are using.

[manishkr]

@manishkr

RTX A4000

Hi
thanks for the pipeline it works well on Xavier NX but when we tried to run the same on RTX A4000 we got above errors

[DaneLLL]

Hi,
On RTX A4000, we would suggest use Video Codec SDK. If you would like to have the gstreamer plugins, please install DeepStream SDK and replace nvvidconv with nvvideoconvert plugins. The nvvidconv plugin is supported only on Jetson platforms.

[manishkr]

Hi @DaneLLL

Failed to stream GPU encoded rtsp-server using gstreamer

Hi I tried running above pipeline on RTX A4000 appsrc name=source is-live=true block=true format=GST_FORMAT_TIME caps=video/x-raw,format=BGR,width=1280,height=720,framerate=15/1 ! videoconvert ! video/x-raw,format=I420 ! nvvideoconvert ! nvv4l2h265enc bitrate=600 ! h265parse ! rtph265pay config-interval=1 name=pay0 pt=96 getting these error logs now 0:00:26.658603717 1 0x7f02f0008000 WARN v4l2 gstv4l2object.c:2991:gst_v4l2_object_probe_caps_for_format:nvv4l2h265enc0:…

this is using nvvideoconvert plugin already

[DaneLLL]

Hi,
It looks like DeepStream SDK is not working properly on the platforms. Are you able to run deepstream-app successfully?

[manishkr]

yes no issue running deepstream-app using sample configs

but getting these warnings

[DaneLLL]

Hi,
Do you use x86 PC with dGPU RTX A4000? If yes, please try the sample:

#include <gst/gst.h>
#include <gst/rtsp-server/rtsp-server.h>
#include <glib.h>
#include <gst/app/gstappsrc.h>

typedef struct {
    GstElement *appsrc;
    guint sourceid;
    guint number_frames;
} CustomData;

static void need_data(GstElement *appsrc, guint length, CustomData *data)
{
    // Here you would typically get your frame data
    // For this example, we'll create a dummy frame
    GstBuffer *buffer;
    GstFlowReturn ret;
    guint size = 640 * 480 * 3;
    guint8 *raw_data = g_malloc(size);

    // Fill raw_data with random values
    for (guint i = 0; i < size; i++) {
        raw_data[i] = g_random_int_range(0, 256);
    }

    buffer = gst_buffer_new_wrapped(raw_data, size);

    GST_BUFFER_PTS(buffer) = data->number_frames * GST_SECOND / 30;
    GST_BUFFER_DURATION(buffer) = GST_SECOND / 30;

    ret = gst_app_src_push_buffer(GST_APP_SRC(appsrc), buffer);

    if (ret != GST_FLOW_OK) {
        g_print("Error pushing buffer: %d\n", ret);
    }

    data->number_frames++;
}

static void media_configure(GstRTSPMediaFactory *factory, GstRTSPMedia *media, gpointer user_data)
{
    g_print ("calling...\n");
    CustomData *data = g_new0(CustomData, 1);
    data->number_frames = 0;

    GstElement *element = gst_rtsp_media_get_element(media);
    data->appsrc = gst_bin_get_by_name_recurse_up(GST_BIN(element), "source");
    g_object_set(G_OBJECT(data->appsrc), "format", GST_FORMAT_TIME, NULL);

    g_signal_connect(data->appsrc, "need-data", G_CALLBACK(need_data), data);

    gst_object_unref(element);
}

int main(int argc, char *argv[])
{
    GMainLoop *loop;
    GstRTSPServer *server;
    GstRTSPMountPoints *mounts;
    GstRTSPMediaFactory *factory;

    gst_init(&argc, &argv);

    loop = g_main_loop_new(NULL, FALSE);

    server = gst_rtsp_server_new();
    g_object_set(server, "service", "8554", NULL);

    mounts = gst_rtsp_server_get_mount_points(server);

    factory = gst_rtsp_media_factory_new();
#if 0
    gst_rtsp_media_factory_set_launch(factory,
        "appsrc name=source is-live=true block=true format=GST_FORMAT_TIME "
        "caps=video/x-raw,format=RGB,width=640,height=480,framerate=1/60 ! videorate ! video/x-raw,framerate=1/60 ! videoconvert ! "
        "video/x-raw,format=NV12 ! x264enc speed-preset=ultrafast tune=zerolatency ! "
        "rtph264pay config-interval=1 name=pay0 pt=96");
#else
    gst_rtsp_media_factory_set_launch(factory,
        "appsrc name=source is-live=true block=true format=GST_FORMAT_TIME "
        "caps=video/x-raw,format=RGB,width=640,height=480,framerate=30/1 ! nvvideoconvert ! "
        "nvv4l2h264enc ! "
        "rtph264pay config-interval=1 name=pay0 pt=96");
#endif

    g_signal_connect(factory, "media-configure", G_CALLBACK(media_configure), NULL);

    gst_rtsp_media_factory_set_shared(factory, TRUE);

    gst_rtsp_mount_points_add_factory(mounts, "/test", factory);

    g_object_unref(mounts);

    gst_rtsp_server_attach(server, NULL);

    g_print("RTSP server is running at rtsp://127.0.0.1:8554/test\n");

    g_main_loop_run(loop);

    return 0;
}

$ gcc appsrc_rtspserver.c $(pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-server-1.0 gstreamer-app-1.0)

We can run it successfully on x86 PC. Please try it on your setup.

[manishkr]

Hi @DaneLLL

pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-server-1.0 gstreamer-app-1.0
-pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0 -I/usr/include/gio-unix-2.0 -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/gstreamer-1.0 -I/usr/include/x86_64-linux-gnu -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -lgstrtspserver-1.0 -lgstapp-1.0 -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0

root@tapway-demo:/opt/nvidia/deepstream/deepstream-6.4/samples/configs/deepstream-app# gcc appsrc_rtspserver.c -pthread -I/usr/include/gstreamer-1.0 -I/usr/include/orc-0.4 -I/usr/include/gstreamer-1.0 -I/usr/include/gio-unix-2.0 -I/usr/include/libmount -I/usr/include/blkid -I/usr/include/gstreamer-1.0 -I/usr/include/x86_64-linux-gnu -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -lgstrtspserver-1.0 -lgstapp-1.0 -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0

root@tapway-demo:/opt/nvidia/deepstream/deepstream-6.4/samples/configs/deepstream-app# echo $?
0

#./a.out
RTSP server is running at rtsp://127.0.0.1:8554/test

output is stuck at that but i port forward 8554 and access from my server, got this

GST_DEBUG=3 ./a.out
RTSP server is running at rtsp://127.0.0.1:8554/test
0:00:08.403444134 17336 0x7fce8c000b90 ERROR rtspclient rtsp-client.c:5168:handle_tunnel: client 0x557551465600: no tunnelid provided

[DaneLLL]

Hi,
It looks like you don’t correctly install DeepStream SDK. Please follow the guidance to remove currently-installed package and re-install it:

Installation — DeepStream documentation