GStreamer mpegtsmux causes pipeline to lock up

Hello,

We are trying to mux an h264 video stream and a KLV stream together using mpegtsmux, but it keeps locking up the stream due to it waiting for data on the KLV pad. I’ve tried to set the KLV stream as a sparse stream, and I’ve tried to send gap events manually within the KLV stream, but the mpegtsmux continues to block.

This is a custom pipeline developed in C.

How can we configure the mpegtsmux so it sends video data without blocking and will mux in KLV data when available?

We are using a Jetson TX2 with JetPack v4.2 and GStreamer v1.16.3.

Thanks.

Hi,
We don’t have experience in the use-case. Would need other users to check and provide suggestion.

Not sure if matroskamux plugin can work with KLV stream. May use the plugin for a try.

Thanks for your reply @DaneLLL. Unfortunately, the matroskamux plugin does not have a meta/x-klv pad template on its sinks.

The documentation on gap events, that I can find, has been lacking; the mpegtsmux is not acting on them as I would expect. How should the timing of the timestamp and duration of the gap event relate to the video buffers coming into the mux and the running time of the mux?

I’m sending them to the mux with the timestamp set to the PTS of the video buffers, and likewise for the duration, though the mux is still blocking.

@DaneLLL I think I found a post on the GStreamer mailing list that speaks to a bug with collectpads which could help explain the weird behavior I’ve seen with mpegtsmux, which uses collectpads in v1.16.3. The linked bugzilla thread has a patch for this bug.

I had tried in past attempts to set the KLV stream as sparse to no avail.

How could I go about compiling gstreamer v1.16.3 from source on the Jetson TX2 with still getting the hardware acceleration plugins? I’ve compiled v1.22.7 on the Jetson TX2 successfully in the past, but that obviously didn’t come with the hardware acceleration features we would like. The hope is if I compile from source with this patch applied it might fix the issues I’ve been having.

You can compile GStreamer as root and copy the hardware plugins to GST_PLUGIN_PATH and it should work, save for composite plugin and I think that’s all.

Having to apply that patch (appears) to not have been necessary. I noticed the poster of the patch had manually sent the stream start event with the GST_STREAM_FLAG_SPARSE flag. Since the source I was using, appsrc, was already sending the stream start event itself, I added a probe to modify the event to include the GST_STREAM_FLAG_SPARSE flag; this fixed my issue with the mpegtsmux blocking on the meta/x-klv pad not having data. As a note, I had the appsrc with the sparse cap set to TRUE prior to trying this, but that seems to have been to no effect.

GstPadProbeReturn app_src_stream_start_probe (GstPad *pad, GstPadProbeInfo *info, gpointer user_data) {
    GstEvent *event = gst_pad_probe_info_get_event(info);

    if (event->type != GST_EVENT_STREAM_START) {
        return GST_PAD_PROBE_OK;
    }

    GstStreamFlags flags = 0;

    gst_event_parse_stream_flags(event, &flags);

    if ((flags & GST_STREAM_FLAG_SPARSE) == 0) {
        if (!gst_event_is_writable(event)) {
            event = gst_event_make_writable(event);

            if (!event) {
                return GST_PAD_PROBE_OK;
            }
        }

        gst_event_set_stream_flags(event, (flags | GST_STREAM_FLAG_SPARSE));
        GST_PAD_PROBE_INFO_DATA(info) = event;

        gst_pad_remove_probe(pad, GST_PAD_PROBE_INFO_ID(info));
    }

    return GST_PAD_PROBE_REMOVE;
}

Hi
For manually building gstreamer framworks, please refer to build instructions in developer guide

1 Like

Hello, are you implying you had this working in 1.22 and not just in 1.16? I copied your example (see minimal test case below) and the muxer never seems to get started. With any luck I’m just missing something obvious.

#include <gst/gst.h>

GstPadProbeReturn app_src_stream_start_probe (GstPad *pad, GstPadProbeInfo *info, gpointer user_data) {
    GstEvent *event = gst_pad_probe_info_get_event(info);

    if (event->type != GST_EVENT_STREAM_START) {
        return GST_PAD_PROBE_OK;
    }

    GstStreamFlags flags = 0;

    gst_event_parse_stream_flags(event, &flags);

    if ((flags & GST_STREAM_FLAG_SPARSE) == 0) {
        if (!gst_event_is_writable(event)) {
            event = gst_event_make_writable(event);

            if (!event) {
                return GST_PAD_PROBE_OK;
            }
        }

        gst_event_set_stream_flags(event, (flags | GST_STREAM_FLAG_SPARSE));
        GST_PAD_PROBE_INFO_DATA(info) = event;

        gst_pad_remove_probe(pad, GST_PAD_PROBE_INFO_ID(info));
    }

    return GST_PAD_PROBE_REMOVE;
}

int main(int argc, char *argv[]) {
   GError *error = NULL;
   GstElement *pipeline, *appsrc;
   GstPad *srcpad;

   gst_init (&argc, &argv);

   pipeline = gst_parse_launch ("videotestsrc is-live=true ! video/x-raw,width=1920,height=1080,framerate=25/1 ! x264enc bitrate=500 ! h264parse ! queue ! mpegtsmux name=mux ! filesink location=test.ts     appsrc name=appsrc is-live=true ! meta/x-klv ! mux.", &error);

   /* Check for errors */
   if (error != NULL) {
       g_printerr ("Error creating pipeline: %s\n", error->message);
       return -1;
   }

   appsrc = gst_bin_get_by_name (GST_BIN (pipeline), "appsrc");
   srcpad = gst_element_get_static_pad (appsrc, "src");
   guint id = gst_pad_add_probe (srcpad, GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM, app_src_stream_start_probe, NULL, NULL);

   gst_object_unref (srcpad);
   gst_object_unref (appsrc);

   gst_element_set_state (pipeline, GST_STATE_PLAYING);

   /* Wait until error or EOS */
   GstBus* bus = gst_element_get_bus (pipeline);
   GstMessage *msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, GST_MESSAGE_ERROR | GST_MESSAGE_EOS);

   /* Free resources */
   if (msg != NULL)
       gst_message_unref (msg);
   gst_object_unref (bus);
   gst_element_set_state (pipeline, GST_STATE_NULL);
   gst_object_unref (pipeline);

   return 0;
}

I was able to compile v1.22.7 on the Jetson TX2, but at the time I couldn’t figure out how to build the hardware acceleration plugins for v1.22.7 so I went back to v1.16. Sorry for the confusion; I got this probe to work in v1.16. Sometime between v1.16 and v1.22.7, I do not know when, mpegtsmux switched from using collectpads under the hood to GstAggregator. I’ve not worked with the newer version of mpegtsmux, but I’ve read the move to GstAggregator solved a number of issues with the muxer.

Running your code myself, I noticed this error in the logs:

DEBUG               GST_CAPS gstpad.c:2732:gst_pad_get_current_caps:<mux:sink_66> get current pad caps (NULL)
DEBUG              mpegtsmux mpegtsmux.c:915:mpegtsmux_create_stream:<mux:sink_66> Sink pad caps were not set before pushing
WARN               mpegtsmux mpegtsmux.c:1010:mpegtsmux_create_streams:<mux> error: Could not create handler for stream

The problem was the caps for the appsrc hadn’t been set, which resulted in the above error. Adding them fixed it for me. I added a queue in-between the appsrc and mpegtsmux, though this might not be necessary.

#include <gst/gst.h>
#include <gst/app/gstappsrc.h>

static GstStaticCaps klv_caps = GST_STATIC_CAPS("meta/x-klv,parsed=(bool)true");

GstPadProbeReturn app_src_stream_start_probe (GstPad *pad, GstPadProbeInfo *info, gpointer user_data) {
    GstEvent *event = gst_pad_probe_info_get_event(info);

    if (event->type != GST_EVENT_STREAM_START) {
        return GST_PAD_PROBE_OK;
    }

    GstStreamFlags flags = 0;

    gst_event_parse_stream_flags(event, &flags);

    if ((flags & GST_STREAM_FLAG_SPARSE) == 0) {
        if (!gst_event_is_writable(event)) {
            event = gst_event_make_writable(event);

            if (!event) {
                return GST_PAD_PROBE_OK;
            }
        }

        gst_event_set_stream_flags(event, (flags | GST_STREAM_FLAG_SPARSE));
        GST_PAD_PROBE_INFO_DATA(info) = event;

        gst_pad_remove_probe(pad, GST_PAD_PROBE_INFO_ID(info));
    }

    return GST_PAD_PROBE_REMOVE;
}

int main(int argc, char *argv[]) {
    GError *error = NULL;
    GstElement *pipeline, *appsrc;
    GstPad *srcpad;

    gst_init (&argc, &argv);

    pipeline = gst_parse_launch ("videotestsrc is-live=true ! video/x-raw,width=1920,height=1080,framerate=25/1 ! x264enc bitrate=500 ! h264parse ! queue ! mpegtsmux name=mux ! filesink location=test.ts     appsrc name=appsrc is-live=true ! queue ! mux.", &error);

    /* Check for errors */
    if (error != NULL) {
        g_printerr ("Error creating pipeline: %s\n", error->message);
        return -1;
    }

    appsrc = gst_bin_get_by_name (GST_BIN (pipeline), "appsrc");
    srcpad = gst_element_get_static_pad (appsrc, "src");
    guint id = gst_pad_add_probe (srcpad, GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM, app_src_stream_start_probe, NULL, NULL);

    GstCaps *caps = gst_static_caps_get(&klv_caps);
    caps = gst_caps_fixate(caps);
    gst_app_src_set_caps(GST_APP_SRC(appsrc), caps);

    gst_object_unref (srcpad);
    gst_object_unref (appsrc);

    gst_element_set_state (pipeline, GST_STATE_PLAYING);

    /* Wait until error or EOS */
    GstBus* bus = gst_element_get_bus (pipeline);
    GstMessage *msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, GST_MESSAGE_ERROR | GST_MESSAGE_EOS);

    /* Free resources */
    if (msg != NULL)
        gst_message_unref (msg);
    gst_object_unref (bus);
    gst_element_set_state (pipeline, GST_STATE_NULL);
    gst_object_unref (pipeline);

    return 0;
}

So the changelog for 1.18 says, but I can’t quite figure out from the documentation how to do sparse streams unfortunately. (It probably doesn’t help that the relevant docs say “WRITEME”.) Gap events aren’t supposed to be relevant for is-live unless I’m misreading things, and while I got something more or less working by experimenting with gap events it was far from ideal. Perhaps I could get it to work but it feels like barking up the wrong tree. To make things more confusing your sparse flag method can be found in a few places in the GStreamer source itself.

This is in 1.16? I never saw any such error in 1.22, nor does your code as given run as desired either. I removed the caps from my minimal test case because it made no difference to the behavior either way, other than showing appsrc as having set them a little earlier or later in the logs.

Note that I mean the muxer stalls waiting for input from the sparse stream, which happens as long as no buffers are pushed and if buffers are not pushed after pushing some to get started. Gap events seem to more or less get around it but from my understanding that’s only supposed to be for sources that aren’t live.

Anyway, thanks for your answer.

Ya, the documentation surrounding this use case isn’t great. I’ve felt a lot of the pain you’re going through.

I’m sorry for not clarifying, yes I ran that code sample I posted with v1.16.

I agree with your assessment that gap events are not an ideal solution. The trick is getting the muxer to respect the sparse stream and not block waiting for data from it. Adding the GST_STREAM_FLAG_SPARSE flag did the trick for me, though if that’s not working for you I wonder if it’s a GstAggregator issue.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.